Inference Is Becoming a Control-Plane Problem
Executive Summary
The important AI story today is not a new model capability but a shift in where capability is won or lost: production inference. Three practitioner accounts from the AI Engineer conference converge on the same point. Once workloads become agentic—variable numbers of calls, unpredictable token lengths, multi-turn state—the serving stack is no longer a plumbing layer beneath the model. It is a control system whose routing, admission, caching, retries, and measurement determine whether the product feels fast, reliable, and economically viable.
What Happened
In Routing LLM Inference in Production: From Engine Signals to Policy, OpenAI engineers Qianru Lao and Lu Zhang describe moving away from a feedback-loop-driven routing approach toward a split control plane and data plane. A global optimizer absorbs demand, capacity and health, network latency, and load-sensitive time-to-first-token and token-generation profiles, then publishes routing weights. Local systems choose synchronously from cached policy.
The significant detail is the objective: not “send a request to the nearest engine,” but minimize expected end-to-end latency. That requires treating cache locality, heterogeneous engine performance, outliers, and retry behavior as parts of one decision. Their account also makes a useful reliability point: retries need bounds, because an overloaded system can turn well-intentioned retries into a storm.
A second talk, Operating Distributed Inference Systems at Scale, reaches the same conclusion from Meta’s operational perspective. Agentic demand does not scale like a conventional service: users differ in calls per task and token consumption, while routing, KV-cache state, batching, scheduling, admission control, and retries affect one another. The proposed operating frame is bluntly practical—avoid, share, move, or delay work—and pairs circuit breakers with queue-depth-based shedding and retry budgets.
The third piece is the methodological check. In Are LLM Performance Benchmarks Reliable?, Google speakers Ashok Chandrasekar and Jason Kramberger argue that a serving benchmark must measure the client harness as well as the server. A nominal 200 QPS test can deliver less actual load when a single-process Python generator is CPU-bound; harness contention can also manufacture latency. Workload choices—sampling and truncation, temperature, stop conditions, cache rate, and multi-turn replay—can make apparently comparable results anything but.
Why It Matters
Together, these are a corrective to the usual model-centric discourse. The question for a deployed AI product is increasingly not just which model scores best, but what a successful task costs at a given reliability and latency target. Meta’s suggested metric—cost per successful task rather than cost per token—captures the difference. A cheap token stream that times out, forces a retry, or breaks a multi-step task is not cheap in the way users experience it.
This reinforces an emerging view of agents: their bottleneck is often orchestration rather than raw intelligence. Better models expand the range of tasks worth attempting; they also amplify load variance and make bad operational defaults more visible. A system that looks impressive in a clean demo can fail economically or ergonomically when long contexts, cache misses, concurrency spikes, and retries arrive together.
The Bigger Story
The mature comparison is therefore not model versus model, but system versus system under a stated workload. That makes benchmark transparency a product concern, not a niche infrastructure concern. Teams should ask what generated the traffic, which requests were replayed, what cache assumptions held, and whether end-to-end success—not just throughput—was counted.
The practical implication is not that every team must build a global optimizer. It is that agent builders should make explicit choices about overload, queues, retry limits, fallbacks, and task-level success criteria before scale forces those choices implicitly. The clearest signal from today’s material is that inference engineering is becoming part of product design.
Further Reading
- Routing LLM Inference in Production: From Engine Signals to Policy — OpenAI’s concrete routing and reliability account.
- Operating Distributed Inference Systems at Scale — Meta’s framing of agentic capacity and task-level economics.
- Are LLM Performance Benchmarks Reliable? — a useful checklist for interpreting serving claims.