A model deployment needs a compatible runtime

DynamoGraphDeployment describes workers but does not establish compatible drivers, GPU discovery, allocation, gang scheduling, networking and monitoring. AICR records those dependencies in a platform-specific recipe with pinned versions, settings, constraints and validation expectations. This article uses a Dynamo 1.4.2 combination, not a universal latest-version recommendation.

The flow snapshots the cluster, generates a recipe, renders Helm or GitOps artifacts, deploys and validates the runtime, then installs the application graph. Offline validation against another snapshot checks declarative constraints and skips behavioral tests. Performance validation separately starts a temporary serving workload and consumes GPUs; it is distinct from basic health checks.

Endpoint selection and transport have different owners

The native Frontend tokenizes requests, scores prefix overlap and load, selects a worker and forwards traffic. It fits a model-specific application endpoint. The Gateway route resolves HTTPRoute to InferencePool and calls Dynamo EPP for selection, while the gateway performs HTTP forwarding. Worker Frontend sidecars use direct mode to avoid rerouting.

The important difference is ownership of transport, retries and timeouts, not a different cache oracle. In the documented version, TCP carries requests and ZMQ carries KV events. Worker runtimes normalize engine events and publish cache state for either routing path.

Request history cannot observe eviction

Remembering which worker received a request predicts cache location but cannot see eviction. Under memory pressure, predicted state can diverge from actual resident prefixes. Live worker events address that mismatch so affinity decisions reflect reported cache contents.

My interpretation is that the useful contribution links configuration artifacts to runtime evidence. Health, DRA, scheduling, scaling and metrics checks have distinct coverage, and ingress selection requires clear platform ownership. Neither a validated recipe nor an offline bundle establishes arbitrary model throughput or proves every network and GPU behavior.

Source: Original engineering article