Faster LLM inference startup: What you need to know
Baofa Fan, DaoCloud / CNCF · 2026-11-06
Starting a container does not immediately add serving capacity. After loading weights, an inference instance may still initialize communication, compile kernels, autotune, and capture CUDA Graphs. This CNCF / DaoCloud overview organizes startup techniques by the state they preserve, making their different recovery boundaries explicit.
Different caches remove different work
Local model-file caches avoid downloads but leave tensor construction and weight transformation. Compilation and autotune caches preserve artifacts without necessarily removing process setup or graph capture. A weight daemon retains processed GPU tensors; a warm instance retains a live engine; a snapshot preserves initialized execution state.
These techniques can combine, but they are not interchangeable. Faster loading can expose imports, communication setup, or graph capture as the next bottleneck. Useful startup measurements end when the restored engine can generate, not when a process appears or files become available.
Weight ownership versus a sleeping engine
vLLM Fast Start uses a persistent daemon that loads and retains GPU weights and shares them with compatible engines through CUDA IPC. Zero-copy engines map existing allocations, so the daemon must outlive them. Configuration checks can reject reuse and fall back to ordinary loading. CUDA IPC is node-local; distributed deployments need the appropriate shards on each node.
SGLang has also merged a weight-cache daemon. Broader model coverage, memory transport, and lifecycle work remain parts of its Fast Engine Recovery roadmap. Separating weight ownership from engine startup does not preserve every other initialization step.
Sleep/wake instead keeps the server process alive while releasing and restoring GPU resources. The article identifies restrictions between vLLM preload’s zero-copy mode and weight offloading through sleep. Compatibility depends on who owns the memory and whether it may be released.
Snapshots and their compatibility envelope
vLLM’s merged experimental initialized-engine snapshot uses CRIU and CUDA checkpoint. Its documented scope is the same machine, a compatible environment, one NVIDIA GPU, and TP=1. Weights and KV state are released before capture and reloaded after restoration, so model files remain dependencies. Restoration also checks a recorded generation result.
SGLang’s initialized snapshot is still an RFC and open implementation PR in the article. External checkpoint-orchestrator proposals divide responsibility: node components capture and store state, while engines know when requests, caches, and communication are safe to capture.
The independent Snapshot project coordinates Kubernetes GPU checkpointing through an operator and node agent; Dynamo integrates it into serving workflows. The described storage path uses a shared PVC, with object storage reserved for future work. Versioned Dynamo v1.4.0 documentation marks GMS plus Snapshot disabled and unsupported for production. Separate features do not imply a supported combination.
Platform workflows are not a shared restore format
llm-d’s GKE Pod Snapshots use gVisor and cloud storage rather than the same format as a CRIU path. Fast Model Actuation can wake a matching sleeping instance or use a launcher with Python modules already imported. ModelExpress transfers weights from a seed replica through NIXL/RDMA, leaving other initialization to the receiver.
Pod checkpointing, GPU proposals, runtime implementations, and OCI packaging are at different stages. The article distinguishes merged CRI interfaces from open implementations and target milestones. Registry distribution does not establish compatibility across runtimes, drivers, or GPUs.
This is a mechanism and implementation-status overview, not a controlled recovery-time comparison. Its practical value is mapping the desired lifecycle boundary—engine restart, resource release, or Pod recreation—to preserved state, remaining dependencies, and version-specific restore constraints.