A model router can eliminate HTTP errors by switching providers while quietly changing business behavior. Sunil Mallya’s LAB article puts latency, availability, and behavioral requirements into one decision framework, encouraging explicit degradation policies per workload. Its useful contribution is a vocabulary for asking whether a task remains acceptably completed after failover.
Original: The LAB Problem: A CAP-Like Tradeoff in LLM Inference · sunil mallya (@sunilmallya) · 2026-10-23 UTC
A successful switch can still fail the business
The opening scenario is hypothetical: the primary model hits rate limits, the router switches to a backup, and latency and errors recover. Later, the team discovers that transactions previously escalated for review are being approved. Request delivery recovered while decision rules changed. This illustrates a risk; it is not a documented incident or experiment.
The backup need not be generally worse. Models can respond differently to prompts, tool formats, ambiguity, and refusal criteria. Provider configurations can also affect nominally identical models. A router optimized only for successful, timely responses may never observe these differences.
LAB means Latency, Availability, and Behavior. Latency asks whether a useful answer meets the task deadline. Availability asks whether work continues during primary-model failure or slowdown, potentially allowing late completion. Behavior asks whether format, decisions, escalation, tool use, and limits satisfy the task. It does not require identical wording; the application defines acceptable rules.
The premise behind the CAP analogy
The author explicitly borrows CAP’s shape, not its mathematical proof. The strongest argument applies when the trusted model cannot answer before the deadline and no tested backup can either. Waiting, refusing, or using an untested alternative cannot then guarantee both a timely answer and acceptable behavior.
A tested backup that can meet the deadline changes that condition. Switching need not necessarily sacrifice behavior. LAB should therefore not be presented as a theorem that every inference system must always choose only two properties. It is a prompt to specify failure assumptions, deadlines, and acceptance criteria before choosing a response.
Latency and availability are not always independent. A chat answer a minute late may be useless, whereas a nightly batch an hour late may remain valuable. Whether delay is permitted determines whether waiting is a viable strategy. Defining availability using the same deadline also makes the two dimensions overlap.
Three positions as policy starting points
The article sketches three choices. LB keeps one tested model without rerouting, favoring speed and behavior on the healthy path while losing availability on failure. AB queues or retries that model, accepting lateness. LA routes to an available substitute, maintaining service and speed while risking behavioral drift.
These labels do not replace a failure state machine. One model can still become slow, so avoiding fallback cannot guarantee latency in every state. Queuing needs deadlines, cancellation, and expiry. Rerouting needs rules specifying eligible tasks. These are implementation implications of the framework, not a complete control algorithm supplied by the article.
Summarization, tool execution, and payment decisions may require different policies in one system. No task automatically belongs to one corner. Its tolerance for delay, wrong decisions, and temporary refusal determines the policy. Workload-specific choices can align degradation with business loss instead of inheriting one default route.
Behavior-aware routing changes more than the URL
The author proposes that the application express the task while a lower layer chooses model-appropriate prompts, parameters, and validation. Backups should pass fixed difficult cases covering ambiguity, tools, refusals, formats, and escalation before receiving traffic. Correctness and rule adherence deserve separate assessment from request success.
An engineering implication is to version routing policies alongside the tested execution conditions. Changing a model, prompt, provider, or serving precision can invalidate earlier qualification. This does not mean every change causes an error; it means permission to handle a workload has evidence and explicit conditions for reassessment.
Validators have limits too. Valid JSON does not establish that a refund should be approved, and a valid tool name does not establish authorization for its arguments. Requirements that cannot be completely checked online may need offline testing, controlled rollout, and human escalation. Those measures reduce risk without proving formal behavioral equivalence.
What failure drills should measure
The proposed drills include rate limits, lost capacity, slow responses, and regional outages. They track timely answers, completed work, fidelity, and correctness separately. That prevents an API success code from being the only evidence that failover restored service. A delayed acceptable response and a fast unauthorized response belong in different outcome categories.
Routing to a smaller model for cost or speed under normal conditions also changes execution conditions. Whether it sacrifices behavior depends on task-specific validation. Lower cost does not necessarily mean worse behavior; an untested route simply cannot support the same promise. Version retirement, moving aliases, and provider changes motivate ongoing checks even without failover.
Tested copies across providers can improve availability while retaining correlated capacity, account, or model-evolution risks. The article supplies neither a quantitative optimization algorithm nor an availability proof or controlled benchmark. Its framework cannot establish a product reliability number.
Read LAB as a design-review outline: document deadlines, behavioral contracts, qualified backups, and failure policies, then identify which switches are tested. The advance is to make application behavior an explicit infrastructure constraint, exposing the distance between a completed request and acceptably completed work.
Make the behavioral contract concrete
A tool task can specify allowed tools, parameters requiring user confirmation, uncertainty that triggers escalation, and whether a late result remains executable. Those constraints are easier to test than requiring a backup to behave “roughly the same.” This is an engineering elaboration of the framework, not a ready-made policy supplied by the article; the task owner must define the contract.