What matters in this issue
Agent serving bottlenecks extend beyond a single model call to how successive requests share caches and how workflows of different lengths share devices. Start with Vesta and AgentWeave. Vesta separates cache ownership from execution placement, explaining why moving state can help under spare capacity yet hurt near saturation. AgentWeave splits long requests along data dependencies, explaining why placing modules on the same GPUs is insufficient on its own. They optimize different layers, so their performance numbers are not directly comparable.
Two further papers add cost and reliability perspectives. TurboProm turns recurring metrics queries into computations over reusable summaries. KedgeFlow connects action approval, single-use authority, and resource execution through an explicit contract. Distinguish the evidence: serving performance from trace replay, estimates of query cost, and authorization-boundary tests against a mock target.
01 · Vesta: stable cache ownership, flexible execution placement
Vesta: Queueing-Aware LLM Scheduling for Real-World Workloads
Lancheng Zou, Bowen Li, Wei Mao, Wenbin Ye, Lecheng Li, Kai Liu, Chao Deng, Jianchen Zhu, Bei Yu · arXiv · 2026-10-10
High cache reuse can still leave requests waiting
Multi-turn agents often append earlier responses and tool outputs to their context. Prefix caching avoids repeated prefill, but routing every turn to the same instance can leave a busy holder queueing while other GPUs sit idle. Moving execution requires transferring KV state. When the entire cluster is congested, that extra work can delay subsequent requests. Vesta adds this downstream queueing effect to placement decisions instead of considering only where the current request would finish fastest.
Each session has a relatively stable holder, assigned by hashing its ID within the current length bucket, that retains its accumulated KV cache. Each request independently receives an executor. A remote executor fetches the reusable prefix and writes newly generated state back to the holder, without offloading another copy into its own host cache. Temporary GPU copies can still be reused; this is not a ban on all replication. The design borrows spare compute without filling long-lived host caches with replicas that displace other sessions.
Reading the diagram: on the left, the holder executes locally. On the right, the executor fetches the orange prefix and writes the new blue state back to the original holder. Moving execution leaves long-term cache ownership intact.
Price the waiting imposed on later requests
A candidate instance’s score combines unfinished prefill backlog, computation and cache-loading costs for the current request, and a congestion penalty for work added relative to the cheapest candidate. With spare cluster capacity, the penalty is small and another executor can help early. Near saturation, it rises, favoring less transfer and recomputation. Compared with SMetric’s migration after the holder becomes overloaded, this makes a per-request decision informed by global load.
When a growing session moves to a larger length bucket, Vesta transfers its longest reusable prefix to the new holder to avoid full recomputation. Hashing balances long-term session counts, not instantaneous request load; executor selection addresses the latter.
Conditions behind the latency gains
The authors replay online agent traces on an H20 cluster with a 200 Gb/s RDMA network. Replay follows original arrival timestamps, serializes dependent turns within sessions, and forces output lengths to match the trace. With 32 prefill/decode-colocated instances running Qwen2.5-7B-Instruct-1M, mean time to first token falls from 3,993.89 ms under SMetric to 1,685.63 ms, a 57.8% reduction. Reported TPS rises from 46.9 to 53.4, approximately 14%.
Gains are smaller with disaggregation. For Qwen in a 24P8D configuration with 2,552 sessions and approximately 33K requests, mean TTFT falls 18.1%. For Hy3-FP8 in a 6P6D configuration, mean TTFT falls 14.5% and P99 falls 29.0%. These belong to different configurations and cannot be combined into one performance result. In the congestion-pricing ablation, the complete system reduces mean TTFT from 2.71 s to 2.17 s on Qwen with 6P2D and 8,030 requests, supporting the contribution of the pricing term.
The combination of cache ownership and queueing costs deserves close reading. Queueing theory supplies a bounded surrogate score, not an exact prediction of real traffic or a latency guarantee. The experiments depend on remotely accessible host caches, offline-calibrated latency models, and substantial long-prefix reuse. Applications that frequently rewrite historical context may benefit differently. Read the holder write-back protocol before the disaggregated experiments and ablations to assess whether queueing is actually the bottleneck in your workload.
02 · AgentWeave: splitting long requests along data flows
AgentWeave: Efficient Distributed Agent Serving via Flow Decomposition
Kaibin Guo, Pengtu Li, Zicong Hong, Zhiyuan Fang, Wuhui Chen, Rachid Guerraoui, Anne-Marie Kermarrec · arXiv · 2026-10-10
Colocating modules still leaves long requests in the way
Placing LLM, retrieval, and reranking modules on separate devices creates a pipeline. When their execution times differ, faster devices wait for slower stages. AgentWeave instead uses an aggregated deployment in which modules execute across all available GPUs, reducing these bubbles. But advancing whole workflows in coarse batches makes short requests wait for long ones and delays the admission of new requests.
The paper therefore refines the scheduling unit from a complete request to execution fragments triggered by data flows. Unlike a change confined to model-internal batching, decomposition operates across workflow modules. It must preserve dependencies: executing early with incomplete data can change the answer.
Not every input can be split
The system first represents a workflow as a flow dependency graph. Decomposable data must form independent subflows whose results are correct subsets of full execution. Documents can be embedded separately, for example, whereas a query expressing a complete intent cannot simply be cut into arbitrary pieces.
Reranking is subtler. Candidate documents can arrive in groups, but global top-k selection requires comparison across groups. AgentWeave adds supplement dependencies that carry accumulated reranking results into the next group, preserving the computation’s complete-input requirement. The important step is specifying how intermediate state continues, rather than assuming that every tool supports chunking.
Reading the diagram: the lower half establishes decomposition rules and profiles their costs offline. The upper half selects chunk sizes from the queue, sends the current batch to the aggregated executor on the right, and returns remaining work for later rounds.
Smaller chunks let new requests enter sooner but increase scheduling and state-management rounds. The online scheduler combines queue state with offline profiles, crediting both completed requests and partial progress while penalizing estimated accumulated overhead. It chooses decomposition granularity for each round; it does not promise to reduce a long request’s underlying computation. This trade-off makes tail latency a separate concern.
Which tasks produce the largest gains?
The authors compare against Ayo and Autellix using the same hardware: four A100 GPUs for RAG and KB-RAG, and two A100 GPUs for DeepSearch and multimodal generation. The main RAG experiments use Qwen3-32B, Qwen3-Embedding-8B, and Qwen3-Reranker-8B. DeepSearch uses Qwen3-14B/4B, while multimodal generation uses Qwen3-14B with Stable Diffusion 3.
Across those four applications, throughput reaches 1.33×, 1.81×, 1.68×, and 1.28× the baseline, respectively. Mean latency falls to 1/2.28, 1/2.45, 1/2.08, and 1/2.15 of baseline latency. The abstract’s largest throughput and mean-latency improvements come from KB-RAG, whereas the largest P90 improvement, 2.04×, comes from RAG. The three maxima are not simultaneous gains delivered to every application.
Against an ablation using aggregation alone, RAG throughput improves a further 1.19× under high load, supporting an additional benefit from decomposition. Accuracy stays at 0.8470 for RAG and 0.8170 for KB-RAG on TriviaQA, and 0.3097 for DeepSearch on GAIA. These are quality results for the evaluated workflows, not proof that arbitrary tools can be decomposed correctly.
This is useful reading for researchers serving complex workflows. Applications currently must explicitly specify decomposable flows and complete-input dependencies; the system does not automatically understand arbitrary agent programs. Offline profiles can become inaccurate when hardware or workloads change. Dynamic production agents, external-tool failures, and multi-tenant deployments remain untested. The paper reports no error bars or significance tests. Start with the RAG decomposition example and the aggregation-only ablation to separate benefits from device placement and scheduling granularity.
03 · TurboProm: reusable summaries for faster metrics queries
TurboProm: Low Cost and High Performance Metrics Observability at Scale
Milind Srivastava, Zeying Zhu, Peilin Xin, Yancheng Yuan, Zaoxing Liu, Vyas Sekar · arXiv · 2026-10-10
Repeatedly scanning similar metrics windows can make observability a substantial computation burden. TurboProm plans summaries from historical queries, precomputes them at ingestion, and reuses SDS objects that identify their data scope, summary algorithm, and payload. Queries without an optimization plan fall back to the original backend, retaining the Prometheus query interface.
Reading the diagram: the Planner selects summaries and query rewrites; metrics enter the Summarizer, compact summaries are stored, and the Query Engine reads them to estimate results.
Validation uses 10 random query workloads, 8K measurements per second, and 20 minutes per workload. The authors summarize approximately halved query cost and one-tenth query latency. Cost is measured using p99 CPU usage and excludes ingestion costs. The 1.2–900× gains for large deployments are extrapolations. Approximate summaries also do not imply zero error for every alert.
The useful contribution is reuse across queries and time windows, rather than the largest speedup. Frequent, recurring aggregations fit this approach better; ad hoc queries still depend on the original backend. Ingestion-priced SaaS products would not automatically turn query-compute savings into lower customer bills.
04 · KedgeFlow: binding executed actions to their approval
KedgeFlow: A Programmable Control Plane for Enterprise AI Agents
Jun He, Deying Yu · arXiv · 2026-10-10
After receiving approval, an agent can still substitute parameters, reuse permission concurrently, or retry after losing a response. KedgeFlow binds signed authority to an exact action. A trusted gateway atomically consumes a shared single-use allowance, rechecks current authority, and dispatches the request. Target-side conditions govern commitment; an uncertain result does not restore the allowance.
Reading the diagram: the left side proposes actions, the middle issues grants and retains authority state, and the right side holds target credentials and verifies execution. Shared state at the bottom prevents repeated consumption of one allowance.
With three etcd members on one host, the authors record 6,560 ledger claim attempts with exactly one success for each of 82 allowances. A separate 32-call gateway race produces one mock-target commitment. The former counts permission consumption, not completed external operations.
This is an execution-contract and boundary-validation study. A permissive policy still authorizes a harmful action. Protection at a mock target does not establish exactly-once effects at real resources. Its value is distinguishing approval, dispatch, and commitment. Production use still depends on credential isolation, enforcement across every access route, and native target-side conditional checks.