Serving the agentic workload: Simon Mo on what AI agents change in vLLM · 2026-11-04
This summary of Simon Mo’s Runtime 2026 talk explains why repeated long prefixes, short outputs and bursty sub-agents make cache placement and routing central to agent serving. It discusses hybrid KV caches, prefill/decode separation and a distributed cache pool. The page explicitly separates talk material, benchmark traces and the summarizer’s additions. Theoretical prefix reuse is not a measured hit rate, and ratios of median input/output lengths are not per-request averages. Performance claims require the linked talk and benchmark configuration.