Run AI Models for Edge Application Development with Just a Few Commands · Joe Halabi, Lin Chai, Maximilien Breughe, Michael Ferry, Charlotte Mallo · NVIDIA · 2026-10-30.
TensorRT Edge-LLM uses a lightweight C++ runtime across development on DGX Spark and deployment on Thor platforms. Its CLI downloads models, builds or reuses matching engines, and offers a local compatible API. The most useful evidence is the conditional DSpark benchmark: a draft model proposes three tokens for the main model to verify.
On DGX Spark 64GB, using Nemotron-3.5 Lightning NVFP4 with thinking enabled, batch size one, one concurrent streaming request, and a 128-token output limit, HumanEval prompt generation rises from 73.4 to 126.9 tok/s. Median TTFT changes from 98.1 to 98.7 ms. SpeedBench 1K/2K generation rises from 73.3/72.8 to 108.5/105.3 tok/s, while TTFT changes from 201.0/329.5 to 203.0/331.6 ms.
Generation speed excludes TTFT. Mean input lengths are 139, 1,153, and 2,247 tokens respectively; peak total memory with DSpark, including the OS, is 27.9GB. This supports a low-concurrency generation benefit, without establishing gains for high concurrency, long outputs, or Thor hardware. The source supplies no suitable core-mechanism architecture figure, so this brief focuses on measurement conditions.