LMCache × Ascend: Multi-Process Mode On Ascend NPUs
LMCache Team · 2026-11-16

LMCache’s multi-process support on Ascend changes ownership rather than introducing another cache-hit algorithm. A standalone lmcache server stores KV state, and vLLM connects through LMCacheMPConnector. A surviving cache service can retain state across worker restarts, while compatible prefixes can be shared by multiple engines. This separates inference-process and cache lifetimes.

Why the process boundary matters

An engine-owned cache typically disappears with the engine. Moving storage, chunking, retrieval and eviction into another process also moves some Python GIL, CPU and I/O work away from scheduling and decode, although necessary connector coordination remains. Lookup and storage run asynchronously where possible, not without any blocking in every case.

The fault boundary is specific: surviving worker failure is not automatic recovery from cache-server failure or replicated multi-node disaster recovery. Reuse still requires compatible prefixes and model configuration. The design separates state ownership; it does not promise that all restarts preserve everything.

Two data paths assign copying differently

The control plane defaults to lightweight ZMQ RPC. In lmcache_driven, the NPU default, a worker exports device-memory IPC handles. The server accesses KV through those handles and executes native multi-layer block-transfer kernels for gather/scatter. Cross-process events signal when transfers are safe; coordination and tensor movement are distinct paths.

Above, the server copies through IPC; below, the worker writes shared L1. Tensor movement is distinct from control messages.View original image ↗

Above, the server copies through IPC; below, the worker writes shared L1. Tensor movement is distinct from control messages.

In engine_driven, workers perform gather/scatter and copy directly into a named /dev/shm pool mapped by both processes, which serves as the server’s L1. Shared memory avoids an extra host-side staging copy, not all device–host movement. This path fits single-format GQA/MHA. DeepSeek-V4’s mixed-dtype latent, rope, DSA-indexer and scale planes use lmcache_driven natively, without host repacking or format downgrades.

Path selection therefore depends on layout and copying responsibility, not an unsupported claim that one is always faster. Hybrid Mamba/GDN state also has alignment and object-group requirements rather than behaving as ordinary interchangeable K/V tensors.

Separate the upstream platform layer from hardware kernels

Device IPC, interprocess events, device operations, multi-plane formats and transfer-mode negotiation move into LMCache’s upstream device abstraction. LMCache-Ascend supplies Ascend C kernels. Compared with runtime patching inside a plugin, this lets the shared control and storage layers evolve upstream while preserving hardware-specific data paths.

Reported validation covers Ascend 910B/910C and Atlas 800I A2/A3, with Qwen, DeepSeek-V4 and GLM-5.3 among the models. Software pairings matter: development branches and releases are not freely interchangeable, and plugin dependencies must match CANN. DeepSeek-V4 needs 4096-token chunk alignment negotiated before sessions start.

Functional validation is not a performance promise

The official verification procedure repeats a long prompt and observes stored tokens, retrieval and external-prefix hits. It establishes a working path, but the article supplies no throughput, TTFT or worker-recovery-time measurements. Disabling internal prefix caching in the example isolates attribution; it is not a universal production recommendation.

CacheBlend and CacheGen remain unsupported in MP mode. Compressed and hybrid layouts for engine_driven and multi-node caching remain future directions. The key advance is reusable state outside engine lifetime and concrete NPU transfer ownership. Stint addresses a different problem: a preempted request’s opportunity to generate after recovery. Persistence and progress are complementary, not substitutes.