Start with ReFlow and CQ2-CARE. ReFlow reuses expensive trajectories while controlling policy lag and GPU assignments. CQ2-CARE shifts coding-context design from making text shorter toward selecting the right files and preserving exactly editable code. Both make the same practical demand: after saving computation or tokens, check whether learning progress and task completion survive.
For inference serving, three KV-storage studies isolate per-object work, partial-chunk amplification on SSDs, and remote-service queues. Hit rate alone cannot establish a benefit. For reliability, ExecWeave separates observation from inference, Diviner puts knowledge updates through replay and human approval, and MCP-FSD examines the gap between tool intent and actual permissions. These provide mechanisms and evidence at different layers, guiding further reading around training, serving, or diagnosis.
01 · ReFlow: reuse recent trajectories while controlling policy lag
ReFlow: Replay-Driven Decoupling for Efficient RL Post-Training
Yukun Tian et al. · 2026-10-15 · Coauthor: Chuan Wu
Waiting in RL post-training often occurs between generation and policy updates. Trajectories are generated token by token, and tool calls add variable pauses; an idle trainer may still lack enough data for its next batch. Asynchrony overlaps the stages, but allowing a larger policy-version gap can increase mismatch with the current model. ReFlow instead reuses recently consumed trajectories, reducing the fresh generation required for each update and adjusting GPU assignments accordingly. Its contribution concerns the lifecycle of training data and the balance between producing and consuming it.
Control reuse and influence separately
Prompts enter a streaming generation queue, and completed trajectories reach the buffer individually. The trainer assembles batches from unused and replayed data without waiting for a complete generation batch. The replay fraction varies with supply and demand, subject to a cap. Separately, the system reduces replayed samples’ relative loss weight and limits policy-version lag. Half a batch coming from replay therefore need not mean half of the update’s influence comes from replay.
Generation streams from the upper left into the age-controlled replay buffer; the trainer below consumes samples while the central scheduler coordinates resources and weight synchronization.
The experiments set both the replay-fraction cap and relative replay loss weight to 0.5, retaining data from the current and previous policy versions. Under assumptions including equal importance-weighted token mass, the paper’s simplified analysis bounds replay’s gradient mixture weight by one-third. This is a local error analysis, not a convergence guarantee for arbitrary tasks. The authors also show that naive fixed-fraction replay can slow learning: reusing data without stability controls does not reproduce the complete design.
Less generation changes the resource balance
Replay reduces generation demand, potentially making a fixed GPU split inefficient. A unified worker prepares both training and inference capabilities and offloads inactive modules to CPU memory. Reallocation changes roles in complete replica groups, preserves existing shards where possible, and transfers corresponding shards to new training replicas. The scheduler uses production, consumption, and idle rates to adjust assignments, while KV occupancy and preemption pressure guide generation concurrency. This avoids repeated cold starts without simply admitting more requests into a congested generator.
Read throughput alongside reward progress
The authors evaluate Qwen3-4B through 32B on up to 128 H800 GPUs against VeRL, one-step-off, and VeRL-async. Training throughput improves by 1.19–2.06× for single-turn DAPO-Math-17K, 1.07–1.92× for multi-turn APPS coding, and 1.31–1.82× for ReTool with Qwen3-8B on 16/32 GPUs. Throughput counts tokens processed by the trainer per step divided by step time, including reused data; it does not mean an equivalent increase in newly generated samples.
Training-reward curves show earlier arrival at comparable rewards in these settings. Role changes on 32 GPUs still take 5.72 or 9.20 seconds. Start with the data cycles and resource rebalancing: together they translate algorithmically permitted reuse into scheduling flexibility. Longer-term generalization, other reward mechanisms, and longer training runs need further evidence; training-reward trajectories do not replace independent downstream evaluation.
02 · CQ2-CARE: select the files to change and preserve editable source text
Skip the Compressor: Coding Agents Fix More Issues When They Pick Whole Files
Xiaocong Yang, Rishit Chatterjee, Himanshu Udupi, Vikram Ramavarapu, ChengXiang Zhai · 2026-10-15
Coding agents working with large repositories commonly retrieve snippets or compress context. Issue repair needs two things simultaneously: the files that should change and their exact original text. Similarity retrieval may spend the budget on related tests, documentation, or callers; token pruning may preserve a filename while damaging the source text a patch must quote. CQ2-CARE prioritizes a few complete files, making localization and editability joint evaluation targets.
From a path inventory to complete files
The model receives the issue and a filtered path inventory, then independently ranks files to modify. The selector takes non-test, non-documentation paths named by two answers, falling back to another ranking when the intersection is empty. The model requests up to four targeted searches for evidence such as definitions or callers, drawing spans from at most eight other files. Selected files and evidence form the context, subject to per-block and total caps; the method does not load an unlimited repository.
Compare retrieval, token pruning, and whole-file selection from top to bottom: they can respectively miss files, damage quoted text, or preserve locatable source. Aggregate resolution rates appear on the right.
Rather than imposing one universal 16K or 32K window, the experiment first constructs CQ2-CARE’s context for each issue and uses its realized token count as that issue’s target budget for all methods. This answers how to organize context at the selector’s operating point; conclusions about arbitrary fixed windows require the additional budget sweep. The model generates one patch, graded by tests. These results do not directly rank multi-turn agents that can reopen files and run tests.
Separate two failure mechanisms
The study uses 300 Python issues from SWE-rebench, each from a different repository, after checking that the gold patch succeeds and the empty patch fails. Resolution is 43.7% with GPT-5.6 and 46.7% with Claude Opus 5, approximately 6.3 percentage points above each model’s best compression baseline, and 9.3 and 12.7 points above its best ordinary retrieval baseline. The trained function retriever SweRank trails by only 0.7 and 1.7 points, suggesting that selecting appropriate, unchanged code admits multiple implementations.
The controls are more informative than the ranking alone. Keeping the selected files fixed but deleting 20% of each line’s tokens reduces GPT-5.6 patch applicability from 98.0% to 78.7% and resolution by 9.7 points. However, when LongCodeZip uses CQ2-CARE’s own candidate files, it resolves 44.7%, close to 43.7%. The evidence favors selecting the right files and preserving complete code lines; it does not establish that compression is universally ineffective.
Implications for context engineering
Agreement between localization answers improves file-level F1, but truncating a single answer to the same number of files produces similar results. Much of the benefit comes from selecting fewer files carefully, rather than agreement alone. With GPT-5.6, the authors estimate full-pipeline API cost at $0.151 per issue, versus $0.168–0.177 for compression approaches’ API calls alone, before compression-model computation.
Read the controlled experiments and discussion first: they turn shorter context into a testable question about whether patches can address real source text. The scope is single-turn Python repair on one benchmark, with one patch sample in most cells. Small gaps are especially sensitive to sampling and possible training-data overlap. Whether multi-turn agents recover compression losses by rereading files must be assessed together with the extra turns and tokens.
03 · ExecWeave: preserve the strength of evidence behind execution links
ExecWeave: Cross-Layer Execution Provenance for Observable Agentic Software
Guo-Wei Wong, Meng Chang Chen · 2026-10-15
Agent tool logs and operating-system events do not inherently share identity. ExecWeave separately collects framework semantics, model/gateway observations, and process, file, and network events, producing an execution graph with provenance, inference, and causal-status metadata. Its heuristic command-to-process bridge requires exactly one compatible observed candidate and abstains when several remain.
Three evidence streams remain separate before normalization and graph construction. The dashed branch adds optional inferred bridges, not directly observed causal relationships.
A crucial negative control shows that uniqueness can still be wrong when the true process is missed and only an impostor remains. At 20 ms polling, the Linux portable collector detects a roughly 16 ms process in only 18/30 trials; median elapsed-time overhead across fixed-work fixtures ranges from 62.3% to 408.8%. Read this as observability that explicitly represents uncertainty, rather than a complete, low-overhead causal tracing guarantee. The single AutoGen–Ollama integration case does not establish cross-framework coverage.
04 · Range: finer reuse without per-object storage work
Range: Decoupling KV-Cache Reuse Granularity from Storage Work
Dingsen Shi et al. · 2026-10-15
Restoring the same 6 GiB of KV can become more expensive when 192 logical objects become 6,144: lookup, allocation, and transfer counts may all grow. Range represents reusable prefixes with range tickets, organizes physical I/O into fixed byte stripes, and retains transfer-bank ownership until asynchronous completion. Attention proceeds when its layer is ready, separating reuse granularity from storage batching.
Tickets on the left identify reusable ranges; banks on the right read and scatter data into KV slots. Each layer waits for readiness, and stores publish only after all required layers finish.
With one H100 NVL, Llama-3.1-8B-Instruct, BF16 KV, and a 65,280-token request containing 49,152 cached tokens, the authors measure TTFT of 1.788–1.803 seconds. Tuned in-process cuFile rises from 2.124 to 3.012 seconds across the plain granularity sweep. The useful design lesson is preserving batch units across the whole path, beyond SSD bandwidth alone. There are only three attempts per cell and one request at a time; multi-rank support is implemented but unmeasured. Concurrent serving and fragmented prefixes limit extrapolation.
05 · LMCache SSD study: predictable hits can still be slower
Characterizing Predictability-Latency Trade-offs of KV-Cache SSD Offloading in LMCache for LLM Serving Systems
Ying He et al. · 2026-10-15
This study compares two runnable deployments: buffered I/O in LMCache 0.3.0 and prefix deduplication plus O_DIRECT in 0.3.16.dev11. Each of four short-context workloads sends 100 stateless requests. For identical inputs, writes fall from 1.62 GB to 11.75 MB, but each of the next 99 requests still reads roughly 12 MB. Mean warm TTFT rises from the archived 27 ms to 93.1 ms.
After the first cold request, repeated requests occupy a narrow latency band; the stable warm path still pays for an SSD read every time.
An approximately 94-token input already matches 80 tokens in the GPU cache, yet its remaining 14 tokens trigger retrieval of the whole partial chunk. The older stack’s low read traffic depends on the OS page cache. This supplies block-device evidence for choosing retrieval versus recomputation after a hit. Version, SSD, and I/O mode change together, so the difference is not a causal ablation of one switch. Inputs shorter than two 256-token chunks also do not represent long-prefix RAG. The paper has no overall architecture figure; its per-request latency plot illustrates the measured behavior.
06 · Diviner: turn diagnostic experience into reviewed, replayable updates
Diviner: Agentic RCA with Human-Governed Adaptation for Large-Scale Infrastructure
Pradipta Ghosh et al. · 2026-10-15
Meta’s Diviner encodes incident knowledge as skills, workflows, and a historical knowledge graph, while a Lead–worker system gathers live evidence. Similar past incidents generate hypotheses, not proof of the current cause. DivinerEval examines saved trajectories for knowledge or tooling gaps; updates require replay and owner approval. The system advises responders and does not execute mitigations.
Encoded experience guides live investigation in the middle; evaluation and approval along the bottom feed updates back into the knowledge base, with human approval before admission.
Holding the model fixed, Diviner + Claude improves Exact RCA over vanilla Claude by 11 percentage points (95% CI: 3–20) and reduces mean wall time by 52% (44–58). The approximately 52% reduction in mitigation time in operational cohorts is a separate observational association, subject to incident mix, staffing, and adoption effects. Read for knowledge governance and evidence contracts. The bundled harness result cannot identify one module’s contribution, and a failed data retrieval is not evidence that a suspected cause is absent.
07 · SpecAgent: draft with a small model, review before state changes
Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents
Zilin Wang, Yinhong Liu, Chao Chen, Qun Liu, Zhijiang Guo · 2026-10-15
SpecAgent lets a small model execute read-only tools and accumulate real observations, pausing before a state-changing action. A strong model reviews the entire segment, retains a semantically valid prefix, and preferably includes a replacement action in its verdict, reducing the cost of generating every step.
Above, the strong model generates every step. Below, the weak model executes reads and holds a write while the strong model accepts valid progress and supplies a repair.
On τ2-Bench Airline, the 4B/397B pairing matches Strong-only Pass@1 at 86.0%, while mean wall time falls from 305.2 to 126.2 seconds. On AutomationBench, however, it reaches 10.5% versus Strong-only’s 15.2%. HBM traffic is estimated from token counts and architecture, not hardware measurements; additional prefill can increase FLOPs. Read it as a cost design around reversible execution boundaries. Benefits require accurate read/write classification, reliable review, and bandwidth-bound strong-model decoding. At large batches, computation can become the new bottleneck.
08 · Remote KV: scale storage alongside context parallelism
When the KV Cache Is Slower Than the GPU: Scaling Remote KV Tiers for Context-Parallel Prefill
Muhammad Ibrahim · 2026-10-15
Muhammad Ibrahim studies remote caching for GLM-5.3 on 8×B300, where fast GPU prefill makes a single Valkey instance a bottleneck. The design assigns chunks to ranks by chunk index, with one remote instance per rank. Ranks agree on an available prefix before loading, preserving collective order despite local failures, alongside asynchronous loading and client bug fixes.
Each prefill rank owns chunks and a Valkey instance, spreading storage work. Decode on the right belongs to the full design; replay latency excludes that handoff.
On a 450-request subset with L2 primed and L1 empty, 98.5% of prompt tokens hit L2; reported prefill p99 falls from 83.32 to 0.67 seconds. The 124× reduction compares a congested path with the complete improved design, not a universal per-request caching benefit. Each condition runs once, replay omits decode handoff, and the selected subset fits in cache. The load-versus-recompute model and failure-consistent cross-rank protocol are the most useful parts to follow.
09 · AMD training co-design: adapt capacity use, kernels, and models together
Architecting for Hardware Diversity: Meta’s End-to-End Co-Design for High-Performance AI Training on AMD GPUs
Haoyu Zhang et al. · 2026-10-15
Meta uses production recommendation models to show why heterogeneous training migration requires more than compatible code. MI350X’s HBM capacity enables changes to batch size, activation checkpointing, and embedding placement. Origami selects GEMM configurations using compute and cache-hierarchy costs, while kernel tuning adapts to multi-chiplet locality and register resources.
Origami’s timing model separates prefetch, overlapped load/compute iterations, and final stores; tile selection must balance parallelism with cache reuse.
Co-design improves one feed-ranking model’s throughput by 1.26×; combined with kernel optimization, it reaches 1.5× over the initial port. Across ten production models at matched GPU counts and batch sizes, the MI350X/B200 throughput geometric mean is 101.2%, with individual ratios of 82%–130%. These results describe a particular deployment and model suite. Emulated TF32 and irregular GEMMs still limit some models. Aggregate parity does not establish equivalent kernels or LLM training performance; read how configuration changes convert capacity into throughput.
10 · MCP-FSD: a tool’s declared purpose does not bound its process permissions
My Credentials, Please: Detecting Filesystem Capability Laundering in MCP Servers at Scale
Zeqian Huang, Mingyue Tang, Aonan Guan, Qiming Sun, Pengzhou He · 2026-10-15
MCP-FSD examines how tool arguments can steer servers into accessing files beyond their declared purpose. It screens tool contracts and path operations, infers least-privilege scopes, plants synthetic credentials outside them, and compares three conditions: an unsandboxed attack, a sandboxed benign call, and a sandboxed attack. This separates exploitability from containment.
The upper pass catalogs and triages servers. The lower pass infers scopes and runs three controls, turning candidates into evidence reports through observed responses and containment outcomes.
Nine registries yield 21,097 server listings, of which 4,806 are triaged. Among them, 787 have filesystem capability flags without a filesystem classification, or 16.4%. That is a screening signal, not a vulnerability rate; dynamic validation confirms 150 instances. The important gap is between tool-level intent and process-level authority. Results depend on successful startup and tested policies. Boundary-inference accuracy itself is unmeasured, and one successful benign call does not establish coverage of every legitimate operation.
11 · LLMEI: trace output failures back to parameter bit flips
Understanding LLM Hallucinations Caused by Soft Errors
Jinghao Wen, Dongning Ma, Ruixuan Wang, Shaohuang Wang, Xun Jiao · 2026-10-15
LLMEI injects faults through integer views and bitwise operations, then observes how outputs change. The study evaluates multiple models on AIME 2024 and LiveCodeBench, sweeping parameter bit-error rates from 10⁻¹² to 10⁻⁴ with ten trials per injection configuration. It groups failures into token corruption, language drift, logical corruption, and premature termination.
The left matches model tensors to injection configurations, the middle generates error maps, and the right applies bitwise changes; output behavior reveals how perturbations propagate.
The authors observe that mild perturbations can leave fluent but logically incorrect output, while severe corruption more often produces meaningless tokens; attention and MLP parameters are generally more sensitive. Its value is connecting hardware reliability to semantic failure diagnostics. This is software-injected parameter corruption: BER is not a production incident rate, and failure-category proportions are not failure rates across all requests. Model size alone does not establish fault tolerance.
Further reading: investigation training and cross-model KV
Scout: train incident investigators with verifiable simulations
Scout 1.0: Post-Training a Model for SRE Investigations
Datadog AI Research · Report: 2026-10-15 · Jonathan Parisot’s post: 2026-10-15
Datadog post-trains GLM-5.3 Flash into Scout. GenSim creates scenarios with known causes from incident mechanisms: CausalGen runs generated applications and injects faults, whereas SynthTel directly synthesizes telemetry, offering different fidelity–cost trade-offs. Rewards combine correctness, exploration, and grounding; training also drops optional tools and varies turn budgets.
One incident blueprint branches into application execution or direct telemetry generation; both produce an alert, queryable evidence, and a reference cause.
AlertEval combines real alerts, archived telemetry, and expert labels, but scores come from an LLM judge’s partial-credit rubric, not an incident-resolution rate. The report places medium-effort investigation quality near GPT-6.1 Sol high and Claude Opus 5.5 high at 3–8× lower estimated inference cost, within its internal evaluation setting. On 90 public SREGym tasks without noise, Scout xhigh completes 80 with one attempt per fault. The key questions concern transfer from simulation to real investigation and rewards that avoid encouraging shallow, overly cautious investigations.
RaReCache: choose which transferred KV positions need recomputation
RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu, Peter A. Beerel · Paper: 2026-10-08 · Sasha Malysheva’s post: 2026-10-15
RaReCache converts a small model’s KV through an offline-fitted linear map, compares full-rank and reduced-rank predictions, and selects tokens relying on directions weakly supported by calibration data. The target recomputes only those positions. For Qwen3-0.6B→14B on the full GSM8K test set, recomputing 30% raises accuracy from pure mapping’s 71.6% to 90.2%, versus 95.1% for the standalone target.
Source KV passes through full-rank and reduced-rank maps; their disagreement selects positions to recompute. The target cache combines mapped and recomputed state.
The maximum 3.04× prefill speedup uses an A100 80GB, mean 149-token inputs, batching, and 20% recomputation, assuming source KV already exists. It is neither the 30% accuracy configuration above nor inclusive of the initial small-model prefill. At light load, framework dispatch can make it slower. Read why the selector outperforms attention weights and how accuracy and serving load jointly determine benefits. Evaluated pairs share a tokenizer within a model family; this is not lossless conversion between arbitrary models.