The LAB Problem: A CAP-Like Tradeoff in LLM Inference

Failover keeps the API up. It doesn’t keep behavior the same.

Sunil Mallya · October 2026

In a high-scale LLM inference system, you can’t guarantee fast answers, constant availability and consistent behavior at the same time. One of them has to give.

I call this the LAB problem: Latency, Availability, Behavior. It’s modeled on the CAP theorem, which says a distributed data store can’t guarantee both consistency and availability when a network partition cuts its nodes off. LAB borrows that shape, not the math. Every production system already decides which property gives way. Most decide by accident.

Here’s what deciding by accident looks like. At 2:14 p.m., your primary model starts returning rate-limit errors. Within seconds, your router shifts traffic to a backup. Errors stop, latency stays green, and on-call closes the alert. Three days later, someone notices the risk agent has been approving transactions it used to escalate.

The service never went down. But the router traded behavior for availability, and nobody chose that.

The router did exactly what it was built to do: keep requests flowing. But restoring service is not the same as restoring behavior. At production scale, six everyday pressures expose that gap:

• Tail latency is unpredictable. A July 2026 benchmark covering 353 models across 44 inference providers found that time to first token jumped from 4.9 seconds at the median to 17.3 seconds at p95, a 3.5× increase. Even healthy inference APIs can blow past latency budgets.
• Provider failures are routine, and backups aren’t immune. A 90-day analysis of 2026 incidents found 20 OpenAI API incidents spanning 72.7 hours and 60 Anthropic API incidents spanning 76.5 hours. These were periods of reported degradation, not necessarily complete outages. The two providers’ incident windows overlapped for just 36 minutes, making multi-provider failover valuable but not foolproof.
• The same model isn’t the same service everywhere. Providers differ in latency, throughput, context limits, pricing and reliability, even when serving nominally identical open-weight models. A 2026 measurement study found that routing decisions must account for the provider, model, task and time, not just the model name.
• Prompts aren’t portable contracts. A prompt tuned for GPT may produce different decisions, tool calls or output formats on Claude. Switching models often requires switching prompts, parameters and validation rules to preserve expected behavior.
• Routers solve reachability, not behavioral equivalence. OpenRouter’s standard model fallback retries the request against another model. It doesn’t automatically substitute a model-specific, optimized prompt. The request may succeed while the application’s behavioral contract breaks.
• Model behavior can drift without a failover. Providers upgrade and retire models, while differences in serving stacks, precision and inference configurations can affect reproducibility. Even an unchanged model name doesn’t guarantee identical behavior over time.

What’s missing is behavior-aware routing. When a request moves to another model, the system should do more than rewrite the destination. It should select the appropriate prompt, parameters and validation rules, then verify that the response still meets the application’s behavioral contract.

A router can keep the API available. It cannot, by itself, keep the application’s behavior dependable.

LAB makes the tradeoff explicit. For each workload, you choose which property gives way, and you design for it.

The three properties

L

Latency: answers arrive on time

A useful response arrives within the task’s deadline. A fast error doesn’t count. Cost usually travels with latency: the faster model is often the cheaper one.

A

Availability: work keeps getting done

Requests still complete when the primary model fails or slows down, even if some finish late.

B

Behavior: output keeps the behavioral contract

Responses meet the application’s required formats, decisions, escalation rules, tool calls and safety limits. That means keeping the contract, not copying every answer the primary gives.

Why separate latency from availability? A chat reply that arrives a minute late is useless. A nightly batch job that finishes an hour late is fine. In strictly real-time APIs, the two overlap.

Pick a position, per workload

Each overlap in the diagram at the top is a position you can pick:

The choice shows up in two modes

CAP only describes failures. Daniel Abadi’s PACELC extended it to normal operation: even with no partition, distributed data stores trade latency against consistency. LAB follows the same pattern, under stress and in normal operation. A rate limit isn’t a network partition, and matching behavior isn’t data consistency, but the design decision has the same shape.

Under stress: availability vs. behavior

Say your primary model is down for 20 minutes, your deadline is 10 seconds, and a new request arrives. You can wait for the tested model and miss the deadline, or reroute to a backup that hasn’t been tested on this task and risk its behavior. Refusing instead of guessing is a stricter version of waiting: it protects behavior, but no useful work gets done right now.

Normal operation: latency and cost vs. behavior

Nothing is down, but you route simple requests to a smaller, faster model to save money and time. That’s the same switch as a failover, made on purpose. If the smaller model hasn’t been tested on that workload, you’ve traded behavior for speed without noticing.

This is a tradeoff, not a hard limit: a cheaper model might behave the same. You only know after testing it.

All the time: behavior drifts

Even if you never switch models, behavior moves under you:

• Versions get retired, and aliases like “latest” move to new models.
• The same open-weights model can be served at different precision or on different software, which can change answers on edge cases.
• Identical requests don’t always return identical outputs, even with fixed settings.

Running copies of the same model on several providers helps: a tested, equivalent copy is how you reach the center of the diagram. But copies only push the problem back. An account-wide limit or a shared capacity shortage can hit all of them, and each copy still drifts.

Choosing positions at scale

A high-scale system doesn’t pick one position. It runs many workloads, and each needs its own:

The point isn’t these specific choices. It’s that each workload gets an explicit, written-down position instead of inheriting whatever the router does by default.

What behavior-aware routing looks like

A plain router changes the endpoint and nothing else. Behavior-aware routing changes everything that should move with the model.

It sits between your app and the router. Your app names the task and its LAB position. The layer picks an eligible model, applies that model’s prompt, parameters and validation rules, and checks the result. A router handles uptime and price. It can’t know what counts as an acceptable underwriting decision.

Measure behavior, not just uptime

Go back to the risk agent from the opening. Every infrastructure metric said the failover worked: valid JSON, on time, zero errors. The one thing that changed, how often it escalated borderline cases, wasn’t on any dashboard.

So track behavior separately, in two ways. Fidelity asks whether a backup keeps the behavioral contract, compared with the primary. Correctness asks whether it gets the right answer, compared with ground truth. A backup can be more correct and still break your format, or match the primary perfectly, mistakes included.

Both are percentages, not yes-or-no. A backup can pass 99% of format checks and still agree on only 90% of key decisions. “99.9% uptime” hides that gap.

Designing for LAB

The goal isn’t to avoid switching models. It’s to make every switch a tested, deliberate decision.

1. Give every workload a LAB position. Write down its deadline, its policy under stress and its policy in normal operation. A summary and a payment decision shouldn’t share fallback rules.
2. Keep a prompt for each model. Version prompts, parameters and validation rules per model. Your app names the task; the routing layer picks the right set for whichever model answers.
3. Test every model before it takes traffic. Backups and cost-saving routes alike. Use a fixed set of hard cases: ambiguity, tool calls, refusals, formats and escalations. Score fidelity and correctness.
4. Run failure drills. Simulate rate limits, lost capacity, slow responses and regional outages. Track on-time answers, completed work, fidelity and correctness separately.
5. Retest on every change. A new model version, prompt, provider or serving setup means a new test. A model that misses the bar shouldn’t take traffic for that workload.

Measure availability as acceptable work completed, not successful API calls.

Every production LLM system already has a LAB position for each workload. The only question is whether someone chose it.