Agents wait between actions for models, tools or people. A permanently assigned Pod ties resources to idle sessions; rebuilding a Pod each turn loses state and adds startup latency. Agent Substrate separates logical sessions from Worker Pods, saving and restoring environment state while retaining Kubernetes scheduling for the pool.
Original: Agent Substrate: Advancing Kubernetes for Agents at Scale
Ekin Karabulut, Omer Yahud · 2026-11-10
Actor state outlives a worker assignment
A separate control plane maps many Actors and environments onto a smaller pool of warm Kubernetes Worker Pods. Idle Actors are checkpointed from gVisor or microVM sandboxes, including memory and filesystem state. Their workers become available. A later request restores the Actor onto a free worker instead of repeating Pod creation through admission, scheduling and kubelet.
The article targets coding assistants alternating repository work and inference, RL rollouts with bursty browser/compiler environments, and short GPU sandboxes for compilation and numerical experiments. Their shared property is persistent environment state without continuous resource occupancy. Kubernetes still controls Worker placement and quotas; sessions are not collapsed into an unisolated process.
The source provides only a decorative header image rather than a useful mechanism diagram. The relevant flow is Actor state, suspension, then restoration on an available worker.
Bound waiting and isolate resource failures
When all workers were busy, a new turn could fail immediately. Router request parking instead retains it and retries restoration, with bounded waiting time and queue capacity. A follow-up prevents requests to already-running Actors from getting stuck behind that queue. Warm capacity still needs backpressure and protection against head-of-line blocking.
Per-container CPU and memory limits inside microVM sandboxes keep a runaway helper from exhausting the whole environment. The article describes stopping the offending container while others continue. That establishes a finer resource-control boundary, not a comprehensive adversarial security evaluation.
GPU support and suspend/restore are still being developed. The design uses Kubernetes DRA, with vendors declaring safe pause operations and device properties required by a snapshot. Restoration matches those properties or falls back to cold start. Restoring an Actor on any free worker must not be interpreted as arbitrary cross-GPU state portability.
Exchange inference and environment signals
Substrate manages where tools run; Dynamo manages model inference. Session identity, tool boundaries, priorities and lifecycle events can help routing preserve reusable KV, avoid hotspots and prioritize interactive work. Dynamo can return projected TTFT and completion time, cache readiness and GPU-pool saturation.
These signals enable coordinated decisions: suspend an environment during a long inference wait, resume it as the response approaches completion, and avoid activating excessive child environments behind a saturated GPU pool. The article describes an integration direction and individual mechanisms. It explicitly states that APIs are changing, GPU support is being rebuilt, and parts of Dynamo integration remain experimental. These should not all be presented as established automatic policies.
Separate density claims from engineering evidence
The introduction claims large scale and higher density, but supplies no complete benchmark methodology for a universal ten-times improvement. That number is therefore not used for comparison here. More concrete evidence concerns the queue fixes, resource limits and compatibility design, including how each changes a failure path.
The AgentOps advance is exchanging lifecycle information between environments and inference. KAI Scheduler still places Worker, training and inference Pods; Substrate performs frequent Actor scheduling inside the pool. Session count is not executable capacity, and suspension does not replace an overload budget.