Two developments make system budgets explicit. FLINT fixes the number of vectors on both sides of retrieval to reduce long-query search cost. The Token-Energy Law separates decode energy into shared weights, KV history and elapsed time, explaining why the same model can have different per-token costs. One supplies a retrieval architecture and efficiency experiments; the other offers an empirical model with a clearly defined measurement boundary.
A second thread concerns stateful agents: AKV tracks cache operations, ClawMMU tracks memory authority, and HarnessAdapt-RL trains tools together with the model. Reliability work spans post-incident configuration changes, alarm attribution, asynchronous detection and invalidation of execution allowances. The useful distinctions are between system measurements, simulation or scripted cases, and protection proposals awaiting evaluation.
FLINT: a fixed vector budget for long-query retrieval
FLINT: Fixed-budget Late Interaction
Himank Handa, Negar Arabzadeh, Matei Zaharia · arXiv · First submitted 2026-10-19
Long queries expose a remaining cost in multi-vector retrieval. ColBERT retains token-level representations for fine-grained matching, making storage and search grow with the input. Its closest fixed-budget comparator, ConstBERT, compresses documents but leaves queries at token granularity. FLINT merits a close reading because it budgets both sides: complex requests retain multiple semantic representations without initiating a separate set of search comparisons for every query token.
After a shared encoder, learned latent queries cross-attend to the complete token sequence, which supplies keys and values. A feed-forward layer and projection refine the pooled representations. Queries and documents share the backbone, pooling module and projection; only their latent sets are separate. Both produce exactly V vectors, scored with MaxSim: take the best document-vector match for each query vector and sum the scores. This compresses encoded representations; it does not truncate a long request to V input tokens.
Read the four architectures from left to right: FLINT pools both queries and documents, then retains multi-vector MaxSim matching.
The budget works at two levels. V vectors per document bound index storage. V vectors per query probe PLAID cells, reducing both the candidate pool and comparisons per candidate. The Transformer still processes the input. Nearly length-independent search therefore does not make encoding free or the complete request strictly constant-time.
The authors train on MS MARCO and evaluate BEIR, CRUMB, TREC Tip-of-the-Tongue 2025 and BrowseComp+. With BERT and V=32, mean response time on TREC’s long queries falls from ConstBERT’s 74.3 ms to 26.1 ms. Search falls from 62.4 ms to 13.5 ms, while encoding remains around 12 ms. Across three long-query corpora, end-to-end gains are 1.6–2.8× with BERT and 1.5–2.7× with ModernBERT. Indices are 5.3–9.3× smaller than token-level ColBERTv2, but approximately the same size as ConstBERT, which also stores 32 document vectors.
Effectiveness involves a real trade-off. BERT FLINT averages 0.452 NDCG@10 on BEIR, below ConstBERT’s 0.469 and ColBERTv2’s 0.502. With ModernBERT, V=32 reaches 0.530, close to ColBERTv2’s 0.534, and surpasses it on CRUMB; ColBERTv2 still leads on the other two complex-query suites. Raising V from 32 to 64 improves complex-query effectiveness but substantially increases search time, with little average BEIR gain. A fixed budget is a tunable compromise rather than a universally lossless change.
For research assistants issuing long, constrained retrieval requests, the contribution is an explicit representation budget at the systems interface. Read the symmetric pooling design and encoding/search breakdown first, then choose V against query length and retrieval requirements. Short queries and small corpora need not see comparable gains, and single-vector retrieval can remain faster. The experiments do not establish improved downstream answer correctness, but offer a testable way to retain fine-grained retrieval while controlling its cost.
Efficiency tests use one A100 80GB, with 2-bit PLAID residual compression and matching retrieval settings for all multi-vector models. After warm-up, each query is timed three times and its median contributes to response-time reporting. This helps isolate architectural differences but is not a competition against every independently optimized retrieval system. The pooling module sees only MS MARCO short queries during training, so long-query results are zero-shot transfer. Budget sweeps cover 16, 32 and 64 only on ModernBERT. Whether different query and document budgets work better remains open.
Token-Energy Law: separate shared weights, KV history and elapsed time
The Token-Energy Law of LLM Decode: Active Bytes and Realized Time Account for Joules per Token
Chenxu Niu, Batuhan Sencer, Wei Zhang, Yong Chen · arXiv · First submitted 2026-10-20
The same model can consume different joules per token as batch size, context and execution paths change. Parameter count or average power alone cannot explain the difference. This paper separates decode energy into three observable coordinates: shared weight bytes per output, private KV bytes read and updated, and elapsed time per useful output. The result is a locally calibrated additive model, not a law multiplying bytes by time.
Weights are shared across requests within an iteration, so larger effective batches amortize their demand. KV histories belong to individual requests and do not shrink in the same way through batching. Iteration traces supply effective batch size and output-weighted history length rather than configured concurrency and maximum context. Accounting thus follows useful tokens when requests finish and batch membership changes.
The panel separates shared weights and private KV demand, adds elapsed time per output, and combines them using locally calibrated coefficients.
The central relationship is:
Here e is GPU-board joules per useful output token; q_w and q_KV are logical weight and KV byte demands; D is decode duration; and T is useful output count. Coefficients are calibrated for the model, precision, hardware, runtime and measurement boundary. The time coefficient is not separately measured idle power. Caching, kernels and auxiliary computation can affect these effective responses; logical demand is not identical to physical HBM traffic.
Experiments cover Qwen2.5-7B, Qwen2.5-14B and Mistral-7B with BF16 weights and KV state, full-history attention, GH200 and GB300, and two-GPU tensor parallelism on GH200. Prefix caching, speculative decoding, chunked prefill, offload and preemption are disabled. Each formal energy run accumulates at least 30 seconds of metered decode and excludes preparation. Across 374 held-out energy executions, condition-mean estimates differ from measurements by at most 3.73%. A separate 177 HBM profiles examine the relationship between byte coordinates and physical traffic. The bound is not a claim about every individual request.
The practical insight is that bandwidth intuition cannot replace elapsed time. Switching between graph execution and eager paths changes how long logical work takes; weight and context estimates alone miss that effect. Conversely, the evaluation uses observed decode duration. Prospective deployment prediction still needs a validated performance model to supply D. The reusable element is the equation’s structure, not universal coefficients across GPUs.
The scope is specific: GPU-board decode energy excludes host CPUs, prefill and datacenter overhead. Two-GPU results do not validate multi-node execution. In MoE, routing determines which requests share expert weights, so activated parameter counts cannot simply be divided by batch size and substituted into the dense-weight term. The contribution to serving research is an interpretable accounting basis with frozen-coefficient tests across workloads. Start with measurement boundaries, changing-request accounting and held-out experiments to judge applicability to a particular service.
Training and held-out energy conditions use disjoint batch–history combinations rather than only repeated timing of one configuration. Departure tests retain the identified model-specific coefficients instead of refitting for each departure pattern. This tests whether the relationship survives workload changes, while remaining distinct from complete online scheduling with bursty arrivals and mixed prefill/decode. It is a useful analytical baseline before validation in more complex serving regimes.
ClawMMU: transformed memory must not promote its own authority
ClawMMU: Protection-Coherent Virtual Memory for Stateful LLM Agents
Jinhao Hu, Bardia Mohammadi, Mofasshara Binte Rafique, Lars Henning Klein, Akhil Arora, Ashvin Goel, Laurent Bindschaedler · arXiv · First submitted 2026-10-20
External text can look like trusted agent memory after summarization, storage and retrieval. ClawMMU attaches owner, scope, origins and authority to each representation, checking mapping into context, derivation into storage, reloading and tool authorization. Summaries inherit the lowest input authority; model-written content remains Evidence unless the owner approves a promotion. Revocation counters live outside checkpoints so restoring old state cannot revive revoked origins.
Four checks surround the context boundary; a payee supplied by an Evidence-level invoice requires owner approval even if the model proposes payment.
The key guarantee concerns argument integrity: values such as a payment recipient must trace to sufficiently authoritative sources. It does not guarantee the model chooses the right action or prevent all disclosure of readable content. This is a mechanism and vision paper; AgentDojo-style attack tests, check latency and approval frequency remain future work. Its contribution is a harness-checkable protection model for memory poisoning, without a measured defense-success claim.
AKV: cache reuse must remember operations as well as tokens
Cache Engineering for Agents
Tianyu Fu, Ruoxi Wang, Zihan Min, Zhongqi Ren, Zhihao Li, Tengxuan Liu, Shengxu Mu, Hanling Zhang, Yu Wang · arXiv · First submitted 2026-10-20
AKV moves context management from text editing to KV operations. drop excludes selected states from future attention; repos rotates retained RoPE keys to new positions without recomputing the full prefix. Dropping A before generating C differs from generating C before dropping A, so identical retained tokens do not imply identical caches. An extended radix tree records operation history; segment-batch attention restores missing states.
All three trajectories retain B and C, but dropping A at different times produces different states; the tree separates their histories.
Across three models and four agent benchmarks, a policy retaining the latest 12 tool responses improves average score by about 1.7 percentage points and reduces estimated cost by 27.7% against high-threshold summarization. GPT-OSS-120B serving uses eight A800 GPUs, four each for prefill and decode; at each method’s peak-throughput concurrency, P95 request latency improves by 1.51×. Costs use token price schedules. Retained states still encode effects of dropped history, so drop is not guaranteed forgetting or privacy deletion.
Post-incident evolution: configuration work needs sustained coordination
Characterizing Post-Incident Software Evolution
Matt Pope, Jonathan Sillito · arXiv · First submitted 2026-10-19
A postmortem action item is not a completed reliability improvement. The study links Wikimedia’s public 2015–2025 incident reports to tasks, reviews and commits, comparing post-incident work with sampled contemporaneous changes. Its 1,302 tasks and 2,203 reviews include both groups; they are not all incident repairs.
Post-incident changes are 4.6× as likely to target server configuration. Resolved tasks have median activity counts of 19 versus 11, reflecting discussion and coordination as well as implementation. Repository composition explains part of the difference. The study adds evidence about what happens after action items are proposed, but one organization, public artifact links and activity proxies cannot establish how much reliability improves. There is no method architecture figure; prioritize artifact linkage and repository-stratified comparisons.
S4-RCA: distinguish mechanism changes from exogenous shifts
What Moved the Alarm Score? Causal Attribution with Exact Computation and Scarce-Incident Estimation
Masaaki Takada, Ryusei Shingaki · arXiv · First submitted 2026-10-20
The largest observed metric shift need not be the best repair target. S4-RCA treats structural-function and exogenous-distribution changes as separate Shapley players, building hybrid reference/incident SCMs to attribute their propagated contribution to an alarm score. For ordered linear SCMs and quadratic scores, first- and second-moment recursion plus quadrature computes the complete attribution without enumerating coalitions. Reference-anchored adaptation estimates changes from scarce incident samples.
Across 30 synthetic 100-variable SCM pairs, function Recall@5 at the largest incident sample size is 0.687, versus 0.460 for parameter-change ranking. Computation reaches 1,000 function/law components in about 11 minutes. Exactness concerns computation given a model, not correctness of the estimated causal structure. Hidden confounding, incorrect ordering and nonlinearity limit applicability. The paper primarily uses equations and experimental plots; prioritize the attribution target and identification assumptions.
BEACON: combine semantics with delayed anomaly propagation
BEACON: Bridging Multimodal Semantics and Asynchronous Propagation for Cloud System Anomaly Detection
Zhe Li, Tengxue Zhang, Yang Shu, Tung Kieu, Lunting Fan, Yingying Zhang, Bin Yang · arXiv · First submitted 2026-10-20
BEACON addresses semantic mismatch between modalities and delayed fault propagation. An information-bottleneck selector combines metric names with temporal statistics, semantic prototypes align logs and metrics, and multi-scale temporal decay aggregates neighboring histories along the dependency graph. An LLM initializes prototypes during training rather than generating every detection online.
Metrics are selected and fused with log semantics, then propagated over delayed graph connections; reconstruction errors yield anomaly scores.
Reported F1 scores are 0.6688, 0.6292 and 0.8723 on AIOps2021, Train-Ticket and anonymized production dataset D1. Improvements over their strongest baselines are approximately 5.9%, 9.0% and 7.9% relative, not percentage points. D1 combines operational telemetry with controlled fault injection; the datasets contain 18, 94 and 14 instances respectively. Results support fusion and asynchronous modeling, subject to topology, training labels and validation thresholds. Identifying an anomalous instance does not itself identify a causal root cause.
Inference configuration: small configurations, diverse combinations
How Developers Configure LLM Inference: A Large-Scale Study of Real-World Practices
Nada Zine, Matias Martinez, Romain Rouvoy, Clément Quinton · arXiv · First submitted 2026-10-20
Using a September 2026 GitHub snapshot, this study identifies 5,398 repositories using Transformers, vLLM or the OpenAI API and extracts 345,117 explicit option occurrences. The pipeline detects interfaces, options and values, then analyzes repository-level adoption so repeated calls do not automatically imply widespread use.
The pipeline confirms inference interfaces before extracting configurations; unresolved values feed back into analysis refinement.
Of 331 documented options, 98.8% appear at least once, but 78.9% occur in at most 10% of repositories using the corresponding backend. Repository-level median configuration sizes are 2, 4 and 4; 56.8–61.6% of multi-option signatures appear in only one repository. Tooling must accommodate both compact common configurations and a diverse tail. The sample covers active public Python repositories with at least 50 stars. Static analysis misses some dynamic values and does not assess whether defaults are appropriate. Popularity is not evidence of optimal performance.
Ternary CPU decode: local weights still leave synchronization costs
Weight Placement Is Not Enough: Synchronization in Multi-Node CPU Decoding of Ternary LLMs
Runguo Li, Dezhen Lu · arXiv · First submitted 2026-10-20 · Reading v2 (2026-10-21)
Here, multiple nodes mean NUMA nodes inside a dual-socket server. Reusing ik llama.cpp kernels and weights, the authors compare row ownership with a Megatron-style column split on Ice Lake and Emerald Rapids. Even after weights are local, synchronization consumes 20–48% of per-token thread time, mostly waiting for late arrivals rather than barrier instructions themselves.
Solid lines are global barriers and dashed lines local barriers; the column split retains local outputs before reading other nodes’ partial sums.
The column split keeps attention heads, KV and partial outputs local. Despite requiring cross-node reductions, it wins all 24 measured configurations against the reduction-free row scheme. In DRAM-bound cases, it is 1.5–1.9× faster than ik at its best thread count. Causes of all late arrivals remain unresolved, shared-machine interference affects some measurements, and accuracy is not token-for-token identity. Waiting-time decomposition explains why counting communication operations alone cannot predict low-bit decode speed.
KPage: compact physical tiles without changing the sparse graph
KPage: Cross-Block Page Compaction for Hardware-Efficient Sparse Attention
Junwei Feng, Qiaorui Chen, Chenyu Liang, Zhen Guo, Jingyi Yang, Xiaoling Zhou, Jiahao Zhang, Dongliang Huang, Yiqiao Liao, Chengru Song · arXiv · First submitted 2026-10-20
Selected sparse-attention blocks can still waste Tensor Core work on padding. KPage splits selected 128-row K/V blocks into 16-row pages and packs across logical boundaries. Loading assembles tiles directly in shared memory without materializing compacted tensors in HBM. Query occupancy selects between two tile shapes; backward uses K/V-owned tiles while preserving original page-pair adjacency.
Pages from partially filled logical blocks share physical tiles while preserving the valid sparse connections.
Across five representative workloads, single-GPU BF16 forward kernels with head dimension 128 achieve a 1.77× geometric-mean speedup over NVIDIA BSA. Padding-free controls average only 1.03×, locating the benefit in fragmentation. Sparse-graph, page-table and backward-layout construction are generally outside timing, so these are not end-to-end model speedups. KPage preserves the supplied sparse pattern’s semantics; it does not establish that pattern’s model quality relative to dense attention.
3D-MoE: complementary expert loads reduce near-memory hotspots
3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP
Yudong Pan, Lian Liu, Shixin Zhao, Yintao He, Kaiyan Chang, Tianhua Han, Chenyang Ai, Haonan Zhu, Mengdi Wang, Yinhe Han, Ying Wang · arXiv · First submitted 2026-10-19
Moving experts near memory does not remove routing skew. 3D-MoE groups experts whose loads offset one another as request domains change, then maps them locally. A CMN broadcast path streams a hot expert’s weight tile once from its host channel to helper cores. Cost-aware remapping addresses longer-term drift.
Offline grouping and placement are on the left, local helper scheduling in the middle, and drift-driven remapping on the right.
For DeepSeek-V2-Lite, Qwen3-30B-A3B and GLM-4.5-Air, cycle-level Ramulator 2.0/BookSim simulation combined with GPU-profiled dense operators yields 4.63× average MoE-layer speedup over EP and 1.58–1.86× end-to-end throughput over the strongest NMP baselines. Estimated extra logic-die area is 2.11%. This hardware/software design depends on a specialized interconnect, stable complementarity and one-time offline optimization; the numbers are not promises for existing GPU deployments.
Expert-layer speed and complete-request throughput remain distinct because non-expert operators also contribute.
Runtime control: approvals need invalidation
Runtime Control for AI Agents: Closing the Evaluation–Execution Gap in Operational Workflows
Jayesh Bhadane, Ramachandra Yerramsetti, Rajesh Gupta, Sumit Kumar · arXiv · First submitted 2026-10-20
An amount, evidence item or authority scope can change between approval and execution. The proposed allowance binds to the facts and rule versions read by the decision. An append-only event log reconstructs state; relevant updates permanently invalidate the old allowance. A final tool-dispatch check is serialized with incoming events.
The controller reconstructs state and returns permission; the agent workflow invokes tools only after checking that permission remains current.
In 42 author-written deterministic replay cases, the system identifies 15 subsequently invalidated approvals, holds 14 affected actions before dispatch and compensates one landed effect. The comparator deliberately does not recheck approvals; human escalations rise from 5 to 14 among 43 proposals. This illustrates temporal control and its cost, not production reliability or superiority to all dispatch-time policy engines. Guarantees depend on complete interception, timely ingestion and complete dependencies; exactly-once external effects do not follow automatically.
TraceLite: compress old observations visually, keep recent evidence as text
TraceLite: Efficient Trace Reduction for SWE Agents via Optical Compression and Redundancy Removal
Jingyu Xiao, Qianxi Liu, Yintong Huo, Yuan Jiang, Michael R. Lyu · arXiv · First submitted 2026-10-20
Inside mini-SWE-agent, TraceLite retains the latest three observations as text, renders older observations at a target 4× compression ratio, removes cross-step duplicates and clears pre-edit observation payloads after code editing. It requires neither compressor training nor auxiliary summarization calls, but depends on a multimodal model reading historical images.
Recent observations stay in a text window while older ones become images; the lower paths remove duplicates and pre-edit payloads.
The main experiment splits SWE-bench Verified’s 500 tasks into 100 for tuning and 400 for evaluation. With GPT-5.6-Luna, high reasoning, temperature 0 and at most 250 steps, it solves 313/400 tasks (78.25%), statistically comparable to the full-trace baseline’s 78.00%. Mean input tokens fall 33.05%, API cost 21.30%, and compression takes 0.438 seconds per observation. The result supports cost reduction with maintained performance, not a demonstrated capability gain of 0.25 points. Image compression can obscure paths, identifiers and error details; cleanup may also require retrieving evidence again.
HarnessAdapt-RL: training produces a model–harness pair
Training the Whole Agent, Not Just the Model
Lingxiao Tang, Wei Tao, Yiwen Guo, Lingfeng Bao · arXiv · First submitted 2026-10-20
HarnessAdapt-RL adapts both policy and execution environment. One frozen teacher derives task-specific mechanism-level repair skills; another modifies tools and hooks for recurring interaction problems. A candidate harness is accepted only when observed pass rate or trainable-rollout rate improves without reducing the other, followed by GRPO on fresh trajectories. This finite-sample rule is not a significance test.
Trajectories inform separate skill and tool changes; selection precedes policy updates, and deployment retains the model and tools but removes training skills and teachers.
With Qwen3.5-9B, deployment removes skills and teachers but retains the final tools. Pass@1 reaches 57.6/37.3/39.7% on Verified/Multilingual/Pro, gains of 9.8/12.6/6.4 percentage points over GRPO. Returning to the initial harness loses up to 7.7 points. Reusing the already discovered final harness in a fresh training run approaches the full method: discovery matters but need not be repeated for every subsequent run. Gains should be considered alongside teacher and comparison-rollout costs.
D-PAL: train drafters against distribution overlap
Beyond the Point Mass: Acceptance-Derived Weights and Distributional Losses for Speculative Decoding
Tianyu Wu, Zhuohan Wang, Jonathan Geuter, Yu Yao, Weifan Jiang, Chengxi Zhang, Ju Li, Minlan Yu · arXiv · First submitted 2026-10-20
D-PAL replaces confidence in one recorded token with target–draft distribution overlap, 1−TV. Expected accepted-prefix length supplies position weights, while KL or Rényi-1/2 can serve as the local loss. Smooth distributional fitting avoids relying only on one sampled label and connects to bounds on acceptance.
The top fits one recorded token; the bottom fits a distribution, with each position weighted by prefix survival and continuation value.
For Qwen3-4B, three draft layers, block size 16, one L40S and one request across six benchmarks, default Rényi raises acceptance length from D-PACE’s 4.50 to 4.56 at temperature 0, and throughput speedup from 3.11 to 3.14. At temperature 1, these move from 4.20 to 4.28 and 2.79 to 2.85. Gains are modest; the abstract’s 4.57 corresponds to KL. The derivation starts from rejection sampling, while the evaluated SGLang backend uses argmax drafts and target-side verification. Training overlap is therefore not literally the runtime acceptance probability.
The matched-weight loss ablations are particularly useful for separating weighting gains from distribution fitting.
KNOX: attested execution does not make semantic judgments infallible
KNOX: Hardware-assisted Regulatory Sandbox for Generative Models
Vasisht Duddu, Adam Caulfield, Xi He · arXiv · First submitted 2026-10-20
KNOX combines a policy manifest, reference monitor and attested execution graph inside Intel TDX and H100 trusted environments. Proposed tool calls are checked against trajectories and guards; accepted and blocked actions enter a hash chain with a topology commitment. OS controls such as seccomp-BPF and Landlock are also required because the monitor cannot inspect every syscall inside a tool.
Planning creates the manifest; the reference monitor and syscall guard mediate execution. Attestation protects records while semantic regulation checks retain model error.
In a 100-case syscall-deviation corpus, the monitor alone prevents 74%; adding the OS sandbox reaches 100%. The regulation guard, however, relies on an LLM judge: it blocks 81.8% of non-compliant flows with the Gemma-3-12B executor, so universal prevention is unsupported. The paper also acknowledges that hardware attestation does not directly bind each installed BPF/Landlock ruleset. Its useful distinction is between untampered execution evidence and correct policy interpretation, with separate trust assumptions for each.
Community: the boundary of DeepInfer’s release
ReignSwords’s discussion, dated 2026-10-21, shifts attention from an open inference engine to what a complete service still requires. DeepSeek-AI’s project documentation lists scheduling, DSpark, SWA bounded replay and structured outputs, while explicitly excluding the KV cache system, VE/prefill/decode disaggregation and cross-replica load balancing. Serving still involves state and traffic management beyond kernels and execution. The post’s code-size, popularity and minimum-GPU claims do not establish deployment requirements and are not developed here.