Provider-Agnostic AI: Swapping Models Without a Rewrite

Provider lock-in in an AI feature does not arrive as a decision. It accumulates. One file imports the SDK directly because it is the fastest way to ship, then a second, then error handling gets written against one provider's exception types, and by the fourth feature the cost of switching is a fortnight nobody has.

That matters more here than in most integrations because the ground moves quickly. Prices change, context windows grow, a model that was clearly best six months ago is now merely adequate, and a cheaper option appears that is entirely sufficient for two of your five call sites.

The fix is small — usually a few hundred lines — and it has to exist before you need it. This is one of the six components that define Template 04 in my stack, and it is the one that pays for itself most reliably.

What actually differs between providers?#

Less than the SDKs suggest, and in four specific places. Knowing which four is most of the work, because everything else is genuinely similar enough to ignore.

Notice what is not on that list. Prompts are portable. Temperature and top-p mean approximately the same thing everywhere. Streaming as a concept is universal. The differences are concentrated in the plumbing, which is exactly the kind of thing an adapter is good at hiding.

What does the interface look like?#

Define it in your domain language, not the provider's. The rest of the application should ask for what it wants, not for a chat completion.

export interface CompletionService {
  complete(input: CompletionInput): Promise<CompletionResult>;
  stream(input: CompletionInput): AsyncIterable<StreamEvent>;
}

export interface CompletionInput {
  messages: Message[];
  tools?: ToolDefinition[];        // Zod schemas, provider-neutral
  maxTokens?: number;
  temperature?: number;
  task: TaskKind;                  // what this call is FOR — drives routing
}

export type StreamEvent =
  | { type: 'token'; text: string }
  | { type: 'tool_call'; name: string; args: unknown }
  | { type: 'done'; usage: Usage }
  | { type: 'error'; error: ProviderError };

The task field is the one people leave out and it is the most valuable. It carries intent — reranking, classification, generation, extraction — which is what lets the adapter route intelligently rather than sending everything to a default. Without it, routing decisions have to be made at every call site, which defeats the purpose.

The StreamEvent union is the second thing worth getting right. A discriminated union means the consumer handles four cases exhaustively and the compiler complains when a new one appears, which is considerably safer than a stream of untyped chunks.

How do you normalise tool calls?#

This is the surface that differs most and the one where a leaky abstraction hurts most, because tool calls are where the model touches your data.

Define tools as Zod schemas on your side. Generate the provider-specific JSON schema from them at call time. When arguments come back, parse them with the same Zod schema regardless of which provider produced them.

That gives you one definition, one validation path, and identical behaviour across providers. It also means the validation layer is written once rather than per provider, which matters because that layer is what stands between a malformed call and your database.

The awkward part is that providers differ in how many tool calls they will emit at once and how they represent parallel calls. Normalise to an array always — a single call becomes an array of one — so consumers never branch on provider behaviour.

How do you normalise streaming?#

Every provider streams, and every provider names its events differently. Some emit content deltas, some emit whole message objects with a delta inside, some interleave tool-call fragments that must be accumulated before they parse.

The adapter's job is to absorb all of that and emit the union above. Consumers should never see a provider-shaped event, and in particular should never accumulate partial tool-call JSON — that reassembly belongs in the adapter, where it is written once and tested once.

One detail that repeatedly causes trouble: providers differ on whether usage figures arrive with the final event or require a separate lookup. Normalise so the done event always carries usage, even if the adapter has to compute it. Downstream cost logging depends on it being reliably present.

How do you handle errors consistently?#

Providers signal the same conditions with different types, codes and headers. Map them to a small set of typed errors that describe what happened rather than which vendor said it.

• RateLimited — carries a retry-after when the provider supplies one. Retryable with backoff.
• ContextTooLong — the input exceeded the window. Not retryable; the caller must shorten.
• ContentFiltered — the provider refused. Not retryable, and it needs a distinct user-facing message.
• ProviderUnavailable — a 5xx or a timeout. Retryable, and the trigger for failover if you have it.
• InvalidRequest — a bug on your side. Not retryable, and it should page someone.

The retryable/not-retryable distinction is the one that earns its keep. Retrying a context-length error wastes money and time and will never succeed; failing to retry a rate limit loses a request that would have worked in two hundred milliseconds.

Each of these also needs a distinct user-facing consequence, and mapping them once in the adapter means every feature gets that behaviour for free. A rate limit should be invisible — retried and served. A context-length error should tell the user their input was too long, ideally with a suggestion. A content filter refusal needs wording that does not imply the system is broken, because it is not.

Getting this wrong produces the most common bad AI user experience: a generic "something went wrong" for five genuinely different situations, three of which the user could have resolved themselves if told what happened.

How do you route by task?#

This is where the adapter stops being insurance and starts paying rent, usually taking a third off the bill.

// One place decides. No call site knows which provider ran.
const ROUTING: Record<TaskKind, Provider> = {
  rerank:     groq,        // latency-critical, no reasoning needed
  classify:   groq,        // short, high volume, cheap
  generate:   anthropic,   // reasoning quality is visible here
  extract:    anthropic,   // structured output, correctness matters
  summarise:  openai,      // whichever measures best this quarter
};

Reranking twenty retrieved passages is a comparison task with no reasoning in it, and it happens on every single query. Sending that to a frontier model is the most common avoidable line on an AI invoice. Generation, where reasoning quality is directly visible in the answer, is worth paying for.

Because routing lives in one map, changing it is a one-line change reviewable in a pull request, and it can be measured — run the evaluation set before and after and you know whether the cheaper route cost you anything.

The measurement is what makes this safe to do aggressively. The intuition that a cheaper model must produce worse results is often wrong for narrow tasks: reranking is a comparison problem where a fast model performs indistinguishably, and classification into five known categories is not where frontier reasoning shows. Without an evaluation set you are guessing, and the guess usually errs toward overpaying.

Route in the other direction too. If the evaluation set shows a cheaper model failing on extraction, that is a reason to route extraction upward rather than to accept degraded output. The point of the map is that both moves cost the same one line.

When should you fail over automatically?#

Rarely, and deliberately. Automatic failover sounds like resilience and frequently produces surprise.

Failover is safe for stateless, idempotent calls#

Reranking or classification can retry against a second provider with no consequence beyond latency. If the primary is down, using the backup is strictly better than failing.

Failover is dangerous for tool-calling flows#

If the first provider already emitted a tool call that executed, retrying against another provider may execute it again. Idempotency keys protect you here, and without them failover is a duplication bug waiting for an outage to trigger it.

Failover changes output quality silently#

A user who gets a noticeably worse answer during a provider incident has no way to know why, and neither does your support team unless the fallback is logged. Record which provider served each call — it belongs in the same table as cost.

My default is failover for the cheap stateless tasks and a clear error for generation, on the grounds that a visible failure is easier to reason about than a quietly degraded answer.

How do you handle retries and backoff?#

Retries belong in the adapter, not at the call sites, because every call site would otherwise implement them slightly differently and none would get the details right.

Exponential backoff with jitter#

Doubling delays without jitter produces synchronised retry storms — every failed request retries at the same instant, hits the same limit, and fails together. Adding randomness spreads them out, and it is the difference between recovering from a rate limit and extending it.

Respect retry-after when it is given#

Providers frequently tell you exactly when to come back. Ignoring that header in favour of your own backoff curve is both slower and more likely to fail again, and it is a common oversight because the header is easy to miss in the error object.

Cap total attempts and total time#

Three attempts and a hard ceiling on elapsed time. A request that has been retrying for thirty seconds has almost certainly lost its user, and continuing to spend on it helps nobody. The ceiling matters more than the attempt count, because a slow provider can blow the time budget in two tries.

Never retry a terminal error#

This is why the typed error classes exist. Retrying a context-length error or a content filter refusal wastes money and cannot succeed. The classification is what makes automatic retry safe to apply everywhere.

Where does the adapter sit in the stack?#

Below your services and above the SDKs, and it should be the only place a provider package is imported. That constraint is worth enforcing with a lint rule rather than a convention, because the fastest path when shipping a feature is always to import the SDK directly.

Above it sit the domain services — a SearchService, an AnswerService, an ExtractionService. Those own prompts, retrieval and business rules. They call the adapter for completions and know nothing about which provider ran or how the response was shaped.

Below it sit the provider packages themselves. Each provider implementation is one file, and adding a new provider means writing one more file that satisfies the same interface and passes the same contract suite. Nothing above the adapter changes.

That layering is also what makes the whole AI capability a module rather than an architecture. The application depends on your interface, and your interface happens to be implemented by somebody else's API.

What about provider-specific features?#

Keep an escape hatch and accept it is not portable. The abstraction is not meant to be pure; it is meant to make the common case swappable.

Prompt caching, extended thinking modes, structured output enforcement, very long context windows — these differ genuinely and sometimes matter enough to use directly. Expose them through an optional field the adapter passes through, and document that any call site using it is pinned to one provider.

The discipline is that such call sites should be few and identifiable. If half your features use provider-specific capabilities, you do not have an adapter, you have an indirection layer that costs maintenance and provides nothing.

How do you test it?#

Run the same suite against at least two providers in continuous integration. This is the only thing that keeps the abstraction honest.

The contract suite#

One set of tests, parameterised over providers. Every adapter must produce the same event sequence, the same typed errors, the same normalised tool-call shape and the same usage figures for equivalent input.

Divergence then appears as a failing build rather than as a discovery on the day you need to switch. Without this, an adapter quietly accumulates assumptions about whichever provider you use daily, and those assumptions are invisible until they break.

Recorded fixtures for the fast suite#

Record real provider responses once and replay them. This gives deterministic tests covering the parsing and normalisation logic without cost or network flakiness, and it is where the bulk of the coverage should live.

A small live suite#

A handful of tests that genuinely call each provider, run on a schedule rather than on every commit. These catch the case where a provider changes its response shape without announcing it, which does happen.

An adapter you have never run against a second provider is not an abstraction. It is a wrapper with aspirations.

What does this cost to build?#

For two providers, roughly two to three days including the contract suite. Adding a third afterwards is typically half a day, because the interface has already absorbed the variability.

Set against that: the last time I moved a production feature between providers, the change was one routing map and a configuration value, and it took under an hour including running the evaluation set to confirm quality had not moved. Without the adapter that same change touches every call site, every error handler and every streaming consumer.

It is also what makes cost routing possible at all, and cost routing alone has paid for the adapter on every project where I have measured it.

There is a second, less obvious return. Because every call passes through one place, the adapter is the natural home for cross-cutting concerns that would otherwise be scattered: usage logging, latency tracking, the per-user spend ceiling, and the rate limiter. Each of those is written once and applies to every feature automatically, including features written after it.

That consolidation is worth as much as the swappability. A cost ceiling implemented at four call sites will be missing from the fifth, and it will be missing quietly, which is exactly the failure mode the ceiling existed to prevent.

Is this over-engineering for a small project?#

It is a fair question, and the honest answer is that it depends on how many call sites you have rather than on how big the project is.

With one AI call in the entire application, an adapter is genuinely unnecessary — swapping providers means editing one file either way. The threshold is around three or four call sites, which most features reach quickly once retrieval, generation and classification are separate steps.

The other consideration is timing. Building it up front costs two days. Retrofitting it across a codebase where provider calls have spread costs considerably more and tends to happen under pressure, because the trigger is usually a price change or an outage.

Conclusion#

Normalise four things: messages, tool calls, streaming events and errors. Define the interface in your domain language with a task field carrying intent. Validate tool arguments with Zod on your side of the boundary. Map provider errors to typed classes that distinguish retryable from terminal.

Then route by task rather than by preference, keep an escape hatch for genuinely provider-specific features, and run one contract suite against two providers in CI so the abstraction stays real.

That is two or three days of work that makes a provider change an afternoon instead of a fortnight, and it usually pays for itself through routing before it is ever needed for switching. Given how fast pricing and capability move, being able to act on a better option quickly is worth considerably more than the abstraction costs.

If you already have provider calls spread across a codebase, the migration is mechanical rather than difficult. Add the interface, implement one adapter for the provider you currently use, then move call sites to it one at a time — each move is independently shippable and independently reversible. Add the second provider and the contract suite last, once the interface has stopped changing shape.