Highlights
Slow agent tools, on-chip distance and edge queues can all bottleneck inference. AOSpec combines speculation with correctness checks; NUNA, DEFRAG and spatial caching adjust computation, communication and placement at different scales. Encrypted split inference provides an adjacent CNN case.
01 · AOSpec: Speculating on actions and observations
AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving
Hao Mark Chen, Jinnan Guo, Wayne Luk, Hongxiang Fan · 2026-08-01
Agents typically generate a tool call and then wait for the environment. Slow tools leave the model idle. AOSpec overlaps model generation and tool execution by speculating on both actions and observations across the agent–environment loop.
Expected Value Decoding prioritizes observations by the latency they could hide, rather than hit rate alone. When an outcome can only be discovered by execution, the system launches a target action in an isolated fork. Joint Action-State Verification checks both the action and its originating state against the committed execution before reuse. On a mismatch, execution follows the real trajectory. Isolation and verification are essential to serial semantics; speculative external side effects cannot simply be committed early.
Reading the figure: compare serial execution, observation-only and action-only speculation, and the combined design. The bottom row executes a slow action in an isolated fork and reuses it only after action and state verification.
The authors replay identical Terminal-Bench 2.0 trajectories across nine harness/model configurations and five generation speeds. Mean latency savings versus serial execution rise from 11.8% at 20 ms/token to 32.5% at 1 ms/token; the strongest practical baseline saves 4.3% and 12.2%, respectively. At 10 ms/token, median and p90 latency fall by 10.7% and 10.5%. Cross-benchmark replay on SWE-bench Verified saves 18.9% at 1 ms/token. These are replay results with modeled generation speeds, not measurements of a live deployment; drafting and isolated execution also consume resources. The central contribution is treating latency and execution correctness together.
02 · NUNA: Accounting for on-chip distance in GPU communication
NUNA: Characterizing and Mitigating Non-Uniform Network Access in Multi-Die GPU Scale-Up Systems
Conor James Green, William Won, Tuan Ta, Bradford M. Beckmann · 2026-08-01
In a multi-die GPU, compute units, memory and inter-GPU I/O ports are not equally distant. NUNA treats this non-uniform network access as an optimization target. NUNA-aware routing selects ports based on physical position, while NUNA-aware placement puts threadblocks and data near the relevant I/O. This extends beyond conventional NUMA placement for compute and memory locality.
In an ASTRA-sim scale-up model, placement alone speeds up collectives by up to 1.5× and placement plus routing by up to 1.8×. Across modeled inference on 12 LLM architectures, time per output token improves by 7% on average and up to 28%. End-to-end results combine inference traces and a roofline model and treat computation and communication serially; they are not measurements on a physical cluster. The practical lesson is that on-chip topology can affect cross-GPU communication, but the gains depend on hardware and parallelization.
03 · DEFRAG: Collaborative retrieval and generation at the edge
From Cloud to Crowd: Democratizing LLM Service with Decentralized Edge Collaboration for RAG
Jiaxing Li, Hengzhi Wang, Feng Wang, Chi Xu, Danyang Song, Ruixiao Zhang, Edith C. H. Ngai, Jiangchuan Liu · 2026-08-02
DEFRAG addresses limited knowledge coverage in edge small language models and the cost and capacity constraints of centralized services. It combines compressed shared knowledge graphs and hybrid retrieval with Bayesian online optimization that selects models and RAG parameters per query, balancing accuracy and cost. Rather than splitting one large model across devices, heterogeneous peers collaborate on retrieval and generation.
The authors evaluate a six-device prototype on 500 question–answer pairs each from Natural Questions and HotpotQA, plus mobile-route, non-uniform placement and domain-specific workloads. In their cost model and comparisons, the accuracy-oriented configuration reaches 67.8% and 81.4% accuracy, narrowing the gap to a 235B model to 6.4 and 2.3 percentage points. With resource sharing they report up to 98.4% lower cost. That reduction depends on the chosen models and pricing assumptions. The contribution is to combine shared retrieval knowledge with per-query configuration choices in edge serving.
04 · Spatial prefix caching: Queueing matters as well as reuse
Spatial Prefix Caching for Wireless Edge LLM Inference: A Stochastic-Geometry and Queueing Framework
Le Yang, Zhouyong Liu · 2026-08-02
Wireless edge nodes differ in both distance and cached KV-prefix depth. The nearest node may miss reuse, while the longest prefix may create communication and queueing hotspots; persistent cache also competes with active requests for GPU memory. This paper models node locations as a Poisson point process and shared prompts as a prefix tree, jointly accounting for association, cache profiles, concurrency and queue stability.
In a normalized simulation at request intensity 4 and a 0.8-second TTFT threshold, coverage is about 0.63 for load-aware routing versus 0.23 for prefix-aware routing that ignores queues. Cache affinity can therefore backfire under load. These are analytical and simulation results under chosen parameters, not measurements of a production wireless network or continuous batching engine.
05 · Encrypted split inference: Choosing device and edge boundaries together
Latency-Optimal Adaptive Split Inference for Privacy-Preserving Cloud-Edge-End Collaboration
Yi Li, Peng Zhang, Man Ho Au · 2026-08-02
This study targets privacy-sensitive CNN batch inference, rather than LLM serving. The device executes the plaintext prefix and retains the key, then encrypts the intermediate activation; edge and cloud run the remaining segments on fully homomorphic ciphertext. A planner jointly chooses the device split and edge termination points, accounting for compute, encryption and communication.
On CIFAR-10 and PathMNIST, the authors report amortized speedups of about 12.9× over full-cloud FHE and 3.9× over block-level splitting. With modeled communication included, latency remains approximately 1,033 and 1,023 seconds per sample respectively. It is a useful adjacent example of heterogeneous inference planning, but the results do not establish interactive latency or transfer to LLMs.
Community
An OpenSRE post describes a self-hosted AI SRE toolkit that gathers logs, metrics, traces and deployment context after an alert, tests hypotheses through tool calls and links conclusions to evidence. The project also describes identifier masking and telemetry opt-out; the post calls it a public alpha. It is worth watching as an incident investigation workflow, though feature descriptions do not establish diagnostic accuracy.