What matters in this issue
Inference efficiency is advancing at several execution layers. TokenGate learns which tokens deserve further computation; Koi learns which deployments work in the current fleet. Cache construction, geographic discovery, driver-level sharing, and cross-model compilation address peak memory, request routing, duplicate residency, and unused compute respectively. Their speedup ratios describe different objects and cannot be added together.
A second thread concerns the basis for action: why an agent keeps checking, how later steps absorb local errors, how model output enters program control flow, and who may authorize a verdict or network connection. These studies make process quality and authority inspectable alongside final success.
Start with TokenGate and Koi. The former offers explicit quality–resource curves and a trainable mechanism. The latter combines heterogeneous hardware and simulation to show how model proposals can participate in a constrained systems feedback loop.
01 · TokenGate: learning which context deserves more computation
TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
Liana Patel, Rishabh Ranjan, Ion Stoica, Matei Zaharia, Carlos Guestrin · 2026-10-19
Long documents and repositories cost more than the KV reads incurred while generating an answer. During prefill, a standard transformer sends every input token through every layer. Training-free pruning can stop some tokens early, but its attention-based heuristics face a difficult trade-off: earlier pruning saves more work while making mistaken removal more consequential. TokenGate is worth reading closely because it learns the selection decision and adapts the model to that decision.
From scores to trainable sparse computation
Lower layers process the full context, the gate selects tokens, and upper layers process the survivors with importance-weighted attention. The right panel shows quality–latency and quality–memory trade-offs.
At a selected layer, a small MLP scores token representations. A top-K operation retains document tokens, alongside separately protected tokens such as the question, answer, or other designated positions. Higher layers process only the survivors and never create upper-layer KV entries for removed positions. Lower-layer computation and state still exist; this is not equivalent to shortening the entire input to K tokens.
The challenge is the discrete top-K operation. The authors incorporate selected gate scores into upper-layer attention weights, providing a gradient path from the task loss through those scores. Per-head scaling and a length-dependent temperature adjust the attention distribution after pruning. Training combines a full-context loss with a loss at a sampled smaller budget: the former supplies learning signals to every token, while the latter teaches compression. Gate and LoRA parameters adapt jointly, rather than attaching a selector to an unchanged backbone.
The setup also defines the scope. The paper places the question before the documents, allowing document representations to contain question information. Task data and adaptation are required; loading an arbitrary model does not establish the same benefit. Ablations show that learned selection beats random selection. Even under random selection, the jointly adapted backbone is more robust, supporting contributions from both token choice and representation adaptation.
What the numbers measure
For Qwen3-1.7B, the evaluation covers code completion, multi-hop QA, summarization, medical extraction, and retrieval. Baselines also use LoRA-fine-tuned models. In BF16 inference at batch size 1 on one A100, operating points matching full-context task quality deliver 1.4–11.3× faster TTFT and 1.5–3.4× lower peak prefill memory. These ranges span tasks, rather than describing a constant speedup for every request. The QA example in the figure uses a different threshold: at 95% of full-context quality, TTFT improves 3.0× and memory falls 2.7×.
The hybrid Qwen3.5-2B experiment also works: on NIAH-32K, a gate at layer 3 gives 3.8× lower TTFT at matched quality. Earlier pruning can be faster but introduces a quality trade-off. A training step is about 1.3× slower, while total training time varies by task; training is not cheaper in every case.
Assessment: an adaptation method for existing models. The valuable contribution is the gradient path connecting pruning to task loss, together with careful baseline implementation. The reported evidence mainly concerns small models, individual requests, and specified tasks. It does not directly establish high-concurrency serving throughput, long-output latency, or quality across arbitrary tasks.
02 · Koi: calibrating heterogeneous inference plans with deployment evidence
Koi: Evidence-Guided, Self-Calibrating Planning for Multi-Cloud, Heterogeneous LLM Inference Fleets
Mankeerat Sidhu, Gangmuk Lim, Hetarth Chopra, Maximillian Eisenberg, Quan Hao Ng, Aryan Gosaliya, Shu Liu, Luka Govedič, Jiaxin Shan, Hongkuan Zhou, Vikram Sharma Mailthody, P. Brighten Godfrey, Ion Stoica · 2026-10-17
Heterogeneous inference fleets face coupled choices: GPU type, tensor and pipeline parallelism, replica counts, and whether admitting a new job justifies changing an existing service. A per-job optimum can consume capacity needed elsewhere. A cheaper instance can also produce more expensive useful tokens when it misses latency targets. Koi puts these decisions into a fleet-wide feedback loop and explicitly limits the LLM’s authority within it.
A deterministic boundary between proposals and execution
Cluster snapshots and telemetry update prediction confidence before candidate generation, scoring, and selection. Deployment feedback informs the next planning cycle.
Every five minutes, the system snapshots jobs, resources, and telemetry. A root planner assigns candidate resource budgets, and specialist agents propose deployment combinations. These proposals cannot directly change the cluster. Deterministic components check memory, parallel partitioning, capacity, quotas, SLO risk, and the number of permitted changes, then predict, score, and jointly select placements. When the search budget expires, the system commits the best feasible plan found, not necessarily the global optimum.
Persistent learning uses a fixed causal graph: configuration decisions influence latency and throughput through observable mediators such as KV pressure, communication cost, and pipeline bubbles. Scoped paths form what the authors call mechanisms. Deployments compare predicted and observed behavior and accumulate supporting or contradictory evidence. Reliable predictions favor exploitation; drift widens uncertainty and encourages renewed exploration. The language model does not validate causal claims simply by writing them down.
The planner also prices loading, concurrent operation, draining, and transient performance loss. One example makes the consequence concrete: reducing an existing service’s replicas to admit a large model would violate its capacity needs, so Koi waits for another batch job to finish. Proposing a plausible configuration and selecting one under global constraints are separate capabilities.
Separate hardware results from simulation
The three-hour, 104-GPU hardware experiment spans three GCP regions with H100s, two A100 memory capacities, and RTXPRO6000s. Against OpenEvolve, the strongest baseline in that experiment, the authors report goodput rising from 3,497.43 to 4,119.51 tokens/s, a 17.79% increase. Cost per SLO-honoring token falls from $0.30×10⁻⁴ to $0.24×10⁻⁴, a 20% reduction. This is not a long-running production validation across multiple cloud providers.
The larger 400/800-GPU experiments are simulations. Goodput improves 45.24% in the three-hour setting but 5.72% in the twelve-hour setting. In the latter, batch progress is 1.10 percentage points below the strongest baseline and cost is 0.59% higher. Koi trades admission, online latency, batch completion, and cost rather than winning every metric. Goodput credits output meeting latency or deadline requirements; planner LLM cost is accounted for separately.
Assessment: a constrained learning controller worth following. Ablations help attribute gains to deterministic tools, persistent evidence, and online updates. However, the fixed causal graph depends on telemetry coverage, and candidate generation can miss useful plans. Convergence under stationary conditions is an assumption; twelve hours does not establish long-term convergence. Read the planning loop and hardware table before the larger simulations to judge maturity accurately.
03 · SigmaTrace: observing GPU-initiated RDMA
SigmaTrace: Library-Agnostic Observability for CPU/GPU-Initiated RDMA Communication
Javid Baydamirli, Didem Unat · 2026-10-16
When a GPU kernel initiates RDMA directly, conventional host-library tracing misses its internal operations. SigmaTrace intercepts driver and mlx5dv boundaries, redirects doorbells to CPU-visible pages, records and forwards them, and recovers WQEs in batches. An offline correlator reconstructs communication across threads, GPUs, and nodes, turning opaque kernel blocks into attributable network activity.
The left path writes directly to NIC doorbells. The right redirects writes to monitored memory, records and forwards them on the CPU, and recovers send-queue information.
Cost depends on trace detail. In a stress test with four A100s sharing one ConnectX-5 and P2P disabled, forwarding alone adds 2–4% per-operation latency; WQE recovery raises this to 22–46%. Simple-mode overhead in the BFS case is 11.5%. Library independence therefore does not imply hardware independence or zero overhead. The doorbell mechanism and tracing levels are the useful reading targets: select detail for the diagnostic question rather than enabling full tracing continuously.
04 · KVCompass: finding prefix caches without creating remote queues
KVCompass: A Discovery Plane for Internet-Scale Prefix Cache Sharing in LLM Serving
Shiyi Ling, Fei Fang, Chen Qian · 2026-10-16
A remote prefix hit need not be faster if its worker has a longer queue. KVCompass separates cache location from index placement. A compact multi-level index records holders and VRAM/DRAM tiers; routing weighs remaining prefill, queueing, and network latency. Local eviction policy remains a worker decision.
Edge nodes look up prefix holders and storage tiers, then route using load and network distance. Worker cache events update the discovery index.
Hardware validation uses 14 RTX 5090 workers across three continents serving BF16 Llama-3.1-8B-Instruct, alongside a 64-worker, eight-region simulation. Mixed-workload throughput on hardware is 28.1 Ktokens/s versus 26.2 Ktokens/s for a central directory. Larger tail-latency gaps often involve baseline hotspots or WAN lookups and must be interpreted with arrival rate. Simulation error against an identically configured hardware run is not quantified. The contribution is discovery and routing, not unconditional intercontinental KV transfer: cross-region replication may ship prompts and recompute instead.
05 · TideKV: fitting the cache-construction peak
TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference
Yu Wang, Zhu Wang, Ning Hu, Yifan Guo, Xiu Ma, Bin Guo, Zhiwen Yu · 2026-10-17
TideKV distinguishes final cache size from the peak required to construct it. It advances the entire prompt through one layer at a time and delays eviction until all chunks finish that layer. Completed layers retain compressed caches. This preserves full causal visibility for selection without retaining full KV for every layer simultaneously.
Only the active layer retains full KV. It processes chunks with complete causal history, compresses at the layer boundary, and passes hidden states to the next layer.
In a matched A100 comparison with BF16 Llama-3.1-8B, a 64K prompt, batch size 1, 2,048-token chunks, and a 4,096-token retention budget, TideKV-Snap and SnapKV both finish with a 512 MiB cache. Peak allocation during construction falls from 22.748 to 16.639 GiB. Savings come from the transient working set, not a smaller decode cache. Hidden-state movement and layer-boundary synchronization add costs; attention backends and allocators also affect process-level peaks. The paper even observes lower allocation but higher NVML peak memory with SDPA.
06 · gKSM: sharing duplicate GPU pages at the driver layer
gKSM: GPU Memory Deduplication at Driver Layer for Serverless LLM Inference
Shuheng Gao, Fan Guo, Yongkun Li, Ziye Yue, Yinlong Xu, John C. S. Lui · 2026-10-18
gKSM moves deduplication into the CUDA driver layer. It intercepts repeated weight transfers during loading and scans stable pages at runtime. Hash matches only nominate candidates; exact content comparison authorizes shared physical pages. Read-only mappings and a kernel/user-space copy-on-write path preserve write isolation. Placement brings reusable instances onto the same GPU because remote GPU reads could cancel the benefit.
Instances retain separate address spaces while identical physical pages are shared. Hashes find candidates, exact comparisons confirm them, and copy-on-write restores private pages on writes.
With four identical Llama3-8B instances on one A100 and 3 requests/s offered per instance, independent vLLM uses CPU offloading under memory pressure and delivers 4.22 requests/s; gKSM delivers 9.07. This is a capacity benefit from avoiding duplicate residency and offloading, not a doubling of single-instance operator speed. Framework transparency does not mean deployment requires no integration: UVM hooks, CUDA VMM, stability detection, and COW must work correctly, and production isolation needs separate assessment.
07 · Conjugate Compilation: optimizing draft and target models together
Conjugate Compilation: Self-Improving Agentic Co-Compilation for Speculative Decoding
Annabelle Sujun Tang, Rohan Mahapatra, Christopher Priebe, Hanyang Xu, Manya Ghobadi, Christopher Rossbach, Hadi Esmaeilzadeh · 2026-10-18
In speculative decoding, independently optimized draft and target kernels may interact poorly on one GPU. This work preserves dependencies while jointly searching multi-stream scheduling, kernel configurations, and compatible cross-model operation groups. It scores complete shared execution rounds. Compiler legality checks, numerical checks, and GPU measurements constrain LLM proposals; accumulated search experience does not mean the LLM’s weights change.
MCTS selects a plan, the LLM proposes refinements, the compiler checks legality, and full-round GPU measurements feed back into search.
For four model pairs on B300, using FP16, batch size 8, 1,024-token inputs, and speculation length at most 6, target-verification throughput averages 1.38× conventional independently compiled sequential execution. Interleaved kernels raise it to 1.47×, with a maximum of 1.59×. Verification throughput differs from full user-facing generation throughput, and search and compilation costs also matter. The useful lesson is to evaluate whole rounds so that locally faster kernels do not produce globally slower execution.
08 · ATEST: allocating reasoning samples using live resource costs
ATEST: Adaptive Resource-Aware Test-Time Scaling for LLM Inference
Mobina Kashaniyan, Ali Jannesari · 2026-10-18
ATEST-Online climbs a sample ladder of 1→3→4→6→8, estimates the value of more sampling from vote stability, and updates GPU energy and latency predictions with recursive least squares. It combines whether further reasoning is useful with how expensive that reasoning is under current load.
Fixed sampling, offline calibration, and online control are compared. Online control reuses answers and combines vote stability with live cost to decide whether to continue.
The main experiment uses Phi-3-mini on an A100, 500 queries per dataset, and four increasing levels of background matrix-multiplication load. Online control defers no queries on any of the four datasets and uses about 92–100% of the budget. GSM8K accuracy is 62.6%, but deferred queries count as wrong, so differences from fixed sampling include coverage effects. Energy charges all GPU power during task occupancy, rather than isolated incremental consumption. Small accuracy gaps also face uncertainty from a single 500-query run. Offline calibration can remain preferable under stable load.
09 · CEMA-Router: routing batches with quality, queueing, and energy
CEMA-Router: Infrastructure-Aware Batch Routing for LLM Inference in Cloud-Edge Systems
Xijia Wang, TianZhang He, Mohammad Goudarzi · 2026-10-19
CEMA-Router estimates answer quality from similar profiled prompts and combines it with endpoint throughput, network conditions, backlog, and energy in a joint batch assignment. Local repair moves a straggling request or swaps two destinations, avoiding a hotspot created by individually attractive choices.
Request monitoring and node/network profiles feed a batch evaluator; evolutionary search and local repairs choose request-to-endpoint assignments.
In the six-endpoint hardware experiment, GSM8K uses batches of 30 requests over ten runs. Relative to UniRoute, completion time falls from 3.351 to 2.443 seconds and estimated energy falls 33.8%. However, quality loss rises from 0.3362 to 0.3670 and routing decision time rises from 0.523 to 1.059 seconds. The improved weighted objective expresses a deployment preference, not simultaneous superiority in quality, latency, and cost. Energy is profile-based, and quality depends on offline examples and retrieved neighbors. This fits services that allow batching and explicit trade-offs.
10 · FaaSpot: protecting structurally costly workflow failures
FaaSpot: Fault-Tolerant Serverless Workflow Execution on Spot Instances
Bandhan Patel, Sushruth Harsha, Kwangsung Oh, In Kee Kim · 2026-10-19
The cost of a spot interruption depends on its DAG position: disrupting a branch before a join can invalidate substantial completed work. FaaSpot identifies structural risk offline, checkpoints by tier, uses survival estimates for scaling, and orders retries by impact. Sequential tasks need not receive the same protection as high-impact positions.
Offline DAG analysis assigns criticality and checkpoint tiers used by online checkpointing, scaling, and retry controllers.
Across eight workflows in a trace-driven emulator, with 500 runs per workflow and method in each of five trials, average success is 89.03% at $0.018 compute cost per successful run. MScheduler achieves 81.78% and $0.040. Tasks run on real Kubernetes, but the interruption timeline is emulated and checkpoint I/O latency is not modeled. A known static DAG and execution-time estimates are additional assumptions. The evidence supports structure-aware fault tolerance, while absolute simulated latency and cost are not cloud deployment guarantees.
11 · MASIF: local failures inside successful traces
MASIF: Implicit Failures for Process Observability in Multi-Agent Systems
Yanwei Huang, Arpit Narechania, Atanu Sinha · 2026-10-19
MASIF studies local errors that do not immediately determine final success, including wrong actions, failed tools, and redundant work. Across 1,828 traces from nine system–task configurations, LLM annotation with human and cross-model checks finds implicit failures in 54.3% of 258 successful traces and 75.2% of 1,570 failed traces. A correct final answer does not establish a healthy process.
Five rows show self-correction, self-workaround, peer repair, orchestrator intervention, and propagation. Crosses mark local errors whose consequences depend on later steps.
Matched replay compares local repair with continuing from the same boundary without changing the original step. Four of seven configurations gain 23.0–46.2 percentage points in additional failure rescue; the others show no clear positive gain. This control matters because rerunning alone can change outcomes. Missed annotations, disagreement over event granularity, the local intervention protocol, and additional judge cost limit generalization. Read this as work on process observability and supervision rather than a universal repair system.
12 · Agentic Interpretation: letting the harness license evidence
Agentic Interpretation: Harness-Governed Evidence Construction for Software Agents
Yifeng He, Yanjun Wang, Siwei Cui, Aritra Sengupta, Murali Krishna Ramanathan · 2026-10-15
AgInt lets the model propose typed evidence records while a deterministic harness decides admission, grades, and whether a verdict is licensed. Records are append-only, and unmet obligations remain visible. A resolvable citation and an execution-confirmed bug witness have different grades; traceability does not imply semantic correctness.
Each column moves from certificate state to a model proposal and a harness decision. Contradictions and unmet obligations cause the commit to be refused.
Experiments separate execution-free patch judgment and fault localization from bug witnessing with sandboxed execution. In the GLM-5 ablation, SWE-bench-Live reject-class F1 rises from 0.268 with a prose certificate to 0.301 with a persistent loop and 0.383 with the reference AgInt configuration. Localization benefits more from the loop itself. Mechanisms therefore matter differently by task, and judgment gains narrow for stronger models. A replayable witness establishes an observed fault; failure to find one does not prove correctness.
13 · Call-site contracts: when model output gains program authority
When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems
Shihao Weng, Longtian Wang, Yang Feng, Mingfei Cheng, Jia Liu, Chao Shen, He Ye, Federica Sarro, Xiaofei Xie · 2026-10-17
This empirical study separates an LLM output’s operation from its downstream consumption. Across 40 systems, 496 call sites, and 917 typed outputs, 48% reach control flow or execution; 40% of those are free text. The question extends beyond parseability to whether code checks or capability boundaries constrain text before it controls behavior.
The study traces output consumers, independently derives expected contracts, extracts implemented contracts, and then screens, validates, and patches reachable mechanisms.
Expected contracts are derived in isolated contexts and compared with constraints extracted from source, with human review and dynamic validation. Of 60 mechanisms tested, 38 trigger. Patches block all original triggers, but only 23 mechanisms are judged completely blocked; a local repair does not establish whole-system security. Selection, contract judgments, and threat reachability limit generalization. The useful audit perspective is to check each boundary separately when one output feeds execution, another model, and a user interface.
14 · Connection Escrow: holding egress until a verifier approves
Connection Escrow: Holding AI-Agent Pod Egress in the Kernel Until an Isolated Verifier Rules
Ashok Subbabhatta Gopalakrishna · 2026-10-19
Connection Escrow redirects connections in the cgroup connect path. A node interceptor retains initial bytes and asks an isolated verifier for approval. Permission binds to one socket, destination, and deadline and can be consumed only once; timeout denies release. This avoids relying on an agent voluntarily honoring an HTTP proxy setting.
The kernel redirects connect, the interceptor retains the socket and identity, and an isolated verifier returns a socket-bound verdict before a destination connection is opened.
One logged experiment passes 36 checks on the eBPF path and 33 applicable checks on iptables. With 16 concurrent agents on a two-vCPU host, median hold time is 32 ms and p95 is 48 ms. The Kubernetes demonstration is also single-node. Connection approval alone does not constrain all subsequent content; the paper adds a separate content gate. Node-root compromise, data attacks on released connections, and covert channels in permitted DNS require separate treatment. The contribution is an explicit enforcement boundary, with evidence still at prototype scale.
15 · ForkDPO: learning proportionate checks at outcome-equivalent forks
Over-Defense in Coding Agents Begins at Silent Forks
Hanjun Luo, Xuecheng Liu, Xiucheng Zhang, Zhuoning Xu, Yingbin Jin, Yankai Chen, Xue Liu, Hanan Salam · 2026-10-18
Once a test passes, repeating it, widening the test scope, or submitting may all receive the same final success reward. ForkDPO trains at these “silent forks”: a rule checker locates excessive actions, forms preferences for proportionate behavior at the same state, and adds counterexamples for premature submission that skips required checks.
The method locates the first excessive action in successful traces and adds skipped-check counterexamples, constraining both unnecessary work and insufficient checking.
After training on 318 tasks, evaluation of Qwen3.5-4B on 48 tasks from four unseen repositories reduces implicit-budget disproportion from 70.3% to 58.9% and median time from 143 to 90 seconds. Oracle pass rate also falls from 94.3% to 93.2%, so the result is not cost-free. Scope is limited by the checker, Python/Go tasks, and a harness with only a bash tool. It does not imply that defensive checks should generally be removed; it supplies learning signals for behavioral choices invisible to final-answer rewards.