What matters in this issue
The central question is what a score actually establishes. FrontierCS 2 requires research patches to satisfy every metric and held-out workload, while CloudIR-Bench shows that diagnosis text can resemble a reference yet identify the wrong cause. These are the two priority readings: they distinguish deliverable capability from aggregate scores in systems research and incident analysis.
The remaining papers follow evidence and state. Failure-attribution research probes internal signals; paired crash tests examine execution understanding; approval and independent-verification studies ask whether evidence shares a common blind spot. KVLearn, UlaCache, HAWK, and RISK-KV address retention, addressing, transfer, and reconstruction, respectively. Koala separates state-ownership handoff from bulk movement. Capacity, throughput, quality, and reliability require different denominators.
01 · FrontierCS 2: requiring research patches to clear every reference threshold
FrontierCS 2: A Benchmark for Computer Science R&D
Mang, Qiuyang et al. · 2026-10-21
Computer-science R&D often means improving a key component inside an already working system while preserving its other performance and quality requirements. FrontierCS 2 turns that setting into 48 tasks across 13 domains. Most tasks start from a paper’s released repository with its central contribution removed; the removed implementation becomes the human reference. Repository exploration, repeated experiments, and final-patch selection become part of the evaluation, extending the familiar problem-statement-to-program format.
Feedback without access to the judge’s evidence
Task construction identifies the contribution, preserves dependencies and declared interfaces, and limits editable files. Reference measurements establish metric thresholds, while the final suite adds unseen seeds, queries, or input graphs. Passing does not require strict improvement in every raw number: one sparse-video-attention task requires latency no greater than 0.96 times the reference, with explicit tolerances for PSNR, SSIM, and LPIPS. A pass means satisfying all these conditions.
Reading the diagram: the agent submits patches and receives development scores through the driver. The evaluator, final suite, and reference scores remain in the separate environment on the right. Development and judging GPUs are separate as well.
Each run has five hours. The agent can submit patches throughout, receiving development scores and sanitized feedback without raw evaluator output or the authors’ code. Its sandbox reaches the network only through the driver. At the end, it must name its own final patch; the evaluator does not select the best development submission on its behalf. Producing a promising intermediate result and reliably choosing a deliverable are different capabilities.
What the 58% result means
The authors evaluate 12 model–harness combinations, with one run per task for each combination. Claude Opus 5.5 with Claude Code passes 58% of runs, and no run passes on 16 tasks. These results measure complete agents, so differences cannot be attributed entirely to the base models. One run also does not establish task-level variability.
Most runs whose aggregate final scores exceed the human reference still miss a threshold on some metric or workload. Average gains can conceal local regressions, especially when quality and latency share a composite score. The paper therefore reports pass rate, score margin relative to the reference, and FCI, which places agent capability and task difficulty on a common scale. These quantities answer different questions.
Start with task construction and the judge. For systems optimization, research agents, and software-reliability evaluation, the useful contribution is an evaluation process that permits exploration while holding the final patch to multiple constraints. Reconstructing a published contribution nevertheless differs from inventing a new research problem. Public papers may also have entered pretraining; network isolation cannot rule that out. The planned private set of more than 100 tasks is outside the denominator of the reported 48-task evaluation.
02 · CloudIR-Bench: similar diagnosis text can still name the wrong cause
CloudIR-Bench: A Benchmark for Root Cause Analysis and Mitigation of Cloud Incidents
Tran, Hoang, Ha, Huong, Dang, Tri, Chan, Jeffrey, Zhang, Hongyu · 2026-10-21
Incident reviews contain symptoms, impact, service relationships, and response history, making them useful for studying textual root-cause and mitigation prediction. Much earlier work relies on private company reports, making comparisons difficult. CloudIR-Bench contributes reusable public data and an evaluation framework, while exposing the gap between text-similarity scores and diagnostic correctness.
Separating retrospective answers from the input
The authors select Azure, Google Cloud, Google Workspace, GitHub, and Atlassian. The final collection contains 950 reports, with 911 root causes and 832 mitigation descriptions. These counts differ: 950 is not the number of fully labeled examples for both tasks. Collection is followed by deduplication, keyword screening, annotation, and consensus review. Two authors with software-engineering expertise independently annotate each report.
Reading the diagram: platforms with sufficiently detailed public information are selected first, followed by report deduplication, screening, and label verification. The final collection emerges from progressively narrower candidate sets.
A key step removes sentences that reveal causes or resolutions from incident summaries, aiming to retain information available at initial reporting. The evaluation also uses GPT-4.1 to condense report information before prediction from the title and summary. This distinguishes copying an answer from a retrospective review from explaining early symptoms, but makes results dependent on whether summarization preserves diagnostic details.
Describing the failure is easier than explaining it
The framework compares eight LLM baselines using BLEU-4, ROUGE-L, METEOR, and BERTScore, generating ten candidates per incident to examine diversity. In-context examples improve mitigation prediction more consistently than RCA; RCA effects vary by platform and metric. Historical examples can suggest familiar actions without establishing that those actions fit the present incident.
The qualitative analysis is particularly useful. Across the three models summarized in its tables, root-cause correctness is 26.4% and mitigation correctness approximately 25.87%. These describe the sampled manual analysis, not a single accuracy estimate for the entire dataset. Even many accepted answers lack technical precision. In one upload-failure case, the model identifies the affected feature but replaces the actual load-balancer configuration change with generic speculation about a backend, API, or database. Shared vocabulary and symptom descriptions can still earn high automatic scores.
The implication extends beyond a leaderboard: RCA must establish whether an explanation supports a concrete response. Evaluating how much an answer resembles an engineer’s report can miss that requirement. Public reports also reveal only what their organizations document, and the annotators were not the original responders. Some cases may lack enough evidence to identify a unique cause.
Prioritize this paper when evaluating diagnosis quality. Read leakage removal, model inputs, and qualitative error categories before comparing automatic scores. This is a textual incident-diagnosis benchmark; it does not directly measure online RCA with logs, metrics, traces, and executable tools. Possible pretraining exposure to public reports remains unresolved.
03 · Failure attribution: routing internal signals into the answer
A Mechanistic Study of Automated Failure Attribution in LLM-based Agentic Systems
Zheng, Wendy et al. · 2026-10-20
Directly prompting an LLM to locate an agent’s failing step can understate the information available in its representations. On Who&When, the authors train linear probes and then use grafting to connect locator heads that attend to failing steps with writer heads that influence the answer. Heads are selected during training; at inference, locator votes determine a step direction injected into writer outputs.
On the Algo subset, Qwen 0.5B reaches 38.10% step accuracy, versus 23.81% for its strongest prompting baseline and 36.51% for probing. This demonstrates a gap between decodable information and generated behavior, not complete causal understanding. Head selection still uses training labels and deployment requires internal activations. Label-free inference is not an unsupervised pipeline. Read the method alongside per-model results, especially the uneven improvements on longer traces.
04 · CATS: compact teams through individual credit
Credit Where Credit Is Due: Contrastive Attribution Topology Search for Multi-Agent Systems
Tritthart, Fabian, Zehle, Tom · 2026-10-21
CATS samples teams from a hand-designed agent supergraph. An attributer compares same-task trajectories and assigns individual credit, which updates edge probabilities and prunes weak edges and unreachable nodes. Roles, models, and tools remain fixed; deployment uses one static team.
Reading the diagram: sampled candidates are executed and compared for attribution; credit feeds pruning, followed by selection of the best candidate from the final round.
With Gemma 4 12B agents and a 31B attributer, three seeds, and 100 held-out tasks per benchmark, BigCodeBench accuracy is 48.0%, versus 42.7% for score-only search, with 54% fewer deployment tokens. Matched candidate evaluations do not mean matched compute, and standard deviations overlap. All three fixed-prompt HotpotQA poisoning runs exclude the poisoned agents, but this is an optimization outcome. Credit is not causally validated and cannot replace tool permissions.
05 · Telecom alarm generation: retaining silent root causes as candidates
Every Alarm Has a Cause: An Open Generator of Telecom Alarm Storms with Exact Root Causes, Bound to NORIA-O, RFC 8632 and TMF642
Mandarapu, Madhulatha, Kunkunuru, Sandeep · 2026-10-21
This work generates synthetic inventory, services, fault propagation, and alarms on real backbone topologies while recording exact causes and propagation paths. Fields bind to NORIA-O, RFC 8632, RFC 8345, and TMF642; unknown terms fail generation. Missing alarms, duplicates, clock skew, and overlapping faults are configurable.
Reading the diagram: Real topology feeds synthetic inventory and faults. Propagation produces both ground truth and an alarm stream; evaluated methods receive only permitted observations.
The interesting result concerns the candidate set. A powered-off site cannot report its own failure, so ranking only alarmed devices systematically misses silent causes. Topology traversal can expose those candidates. The simple earliest-critical-alarm rule is also strong across generated configurations. The real-PCIC comparison uses labels derived from a type-level causal graph, unlike the synthetic device-level task, so accuracy rates are not directly comparable. This is useful for studying missing evidence and evaluation bias; exact synthetic labels do not establish production realism.
06 · KVLearn: retention decisions combine reuse and resource costs
To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems
Liu, Dong, Yu, Yanxuan, Jiang, Eric, Wang, Shu, Wu, Ying Nian · 2026-08-10
KVLearn adds a retention control plane to disaggregated prefill/decode serving: a small MLP predicts prefix reuse, a score combines recomputation, transfer, and occupancy costs, and a feedback controller adjusts admission with pool pressure. Admission and eviction are considered together so inexpensive, low-value prefixes do not displace costly state.
Reading the diagram: prefill produces KV on the left, the middle control plane decides retention, and decode consumes it on the right. Hit rate and pool pressure feed subsequent decisions.
The paper’s Small configuration uses eight A100 GPUs, a 256 GB pool, and 200 Gbps networking. Its MM-Session table reports 38.2 ms TTFT versus 52.5 ms for Mooncake-LRU. These are workload-replay results under the listed conditions. Some workload sizes and transfer figures differ between prose and tables, and the cost derivation leaves tension around cancellation of transfer hops; the results do not justify a general optimality claim. The source identifies August 10 as its first version, rather than a new release on this issue’s date.
07 · SYSRB: test the non-crashing counterpart too
Do Coding Agents Understand Why Systems Crash—and Why They Don’t?
Huang, Chenxi et al. · 2026-10-21
SYSRB tests execution reasoning with 934 paired Linux and OSS-Fuzz source trees. Each pair differs by one commit; the same input crashes one tree and not the other. Agents can search source code but cannot compile or run the reproducer. Comparing crash-prediction probabilities across both trees prevents suspicious-code recognition alone from earning a strong result.
Reading the diagram: The same input is paired with trees differing by one commit; differences in predictions help isolate reasoning beyond suspicious-code recognition.
With eight samples per version and up to two hours per run, the strongest combination has 72.5% true-positive and 4.8% false-positive rates, still missing over a quarter of real crashes. Finding decisive code does not guarantee understanding its effect on a distant crash site. This isolates source reasoning and retrieval; it is not a complete repair evaluation with dynamic debugging available.
08 · MicroPerf-Bench: optimization must improve the deployed service
Benchmarking Coding Agents for System-Level Performance Optimization
Zhang, Yanze et al. · 2026-10-21
MicroPerf-Bench constructs 65 optimization instances across seven microservice applications. Controlled anti-pattern injection establishes attributable degradation, followed by container rebuilds, functional checks, and mixed-workload measurement. Scores measure recovery of injected latency loss; adding hardware, disabling tracing, or weakening transaction safety does not count as a valid optimization.
Reading the diagram: Anti-pattern and application selection precede injection, validation, and removal of leakage cues; functional and performance checks constrain each instance.
Nine model–harness combinations run three times each, yielding 1,755 trials. Mean recovery is 34.8/100 and full recovery occurs in 16.7% of trials. Many zero-score patches compile and preserve functionality but do not reduce end-to-end latency. Separating bottleneck identification from code generation is the key contribution. Results depend on injected patterns and fixed workloads; the additional profiling arm is a single run and should not be treated like the three-run means.
09 · D-Pingmesh: probes follow network reconfiguration
D-Pingmesh: Dynamic Probing and Fault Localization for Reconfigurable Datacenter Networks
Wei, Yunze et al. · 2026-10-21
In a reconfigurable network, the same probe can follow different paths before and after a control-plane update. D-Pingmesh uses topology, forwarding weights, and hash information to select probes that improve interface coverage. After changes, it replays the old list and repairs gaps instead of rebuilding everything. Rho-EM jointly estimates old/new-path mixtures and the fault set.
Reading the diagram: Control-plane rules inform probe generation, incremental updates, and boundary localization; host probe results feed back into diagnosis.
On an eight-pod topology with a 0.99 coverage target, the authors report 17.7 times fewer probe tasks than Pingmesh. In ns-3 boundary-window experiments, micro F1 rises from deTector PLL’s 0.871 to 0.953. The contribution is interpreting evidence across configuration changes, rather than merely sending more pings. Deployment depends on trustworthy snapshots and path resolution; simulated accuracy is not a production localization rate.
10 · UlaCache: storing more distinct sparse-attention history
UlaCache: Unified Logical Addressing for Sparse LLM Serving
Zhu, Isaac et al. · 2026-10-21
Sparse attention reads a subset of KV at each step but must retain history that later selections may need. UlaCache connects GPU and CPU storage through stable logical addresses, distributes history across TP ranks, and manages a local shared pool in DP. PB-LRU batches recency updates with buckets and bitmaps; mappings change only after data transfers complete.
Reading the diagram: DP manages local GPU/CPU pools on the left; TP aggregates distributed history on the right. Mappings and local buffers connect logical references to physical storage.
In GLM-5.3 decode experiments with 64K–128K inputs, peak throughput on eight B300 GPUs in TP without speculative decoding is 3.51 times SGLang’s. Each method contributes its own peak, and all prefill finishes before decoding starts; this is neither a same-batch comparison nor whole-request acceleration. At batch size one it remains slower than GPU-resident SGLang, exposing the lookup, scheduling, and communication costs of greater capacity.
11 · HAWK: KV transfer must respect destination memory layout
HAWK: Heterogeneity-Aware KV Cache Transfer for Efficient GPU-NPU Phase-Disaggregated LLM Inference
Park, Junhyeok et al. · 2026-10-21
HAWK assigns prefill to GPUs and decode to NPUs, confronting different KV layouts and direct-communication interfaces. It allocates tensor-layout-aware RDMA buffers per NPU DRAM channel, then coordinates GPU-side transformation and transfers to reduce host staging and post-transfer cross-channel rearrangement.
Reading the diagram: GPU prefill produces KV, which reaches NPU decode through layout handling and direct transfer while each side retains a compute-appropriate layout.
The authors evaluate an A100/Tenstorrent p150a system. At 90% SLO attainment, TTFT-supported request rates differ from the GPU direct-transfer configuration by 2.90% for chatbot and 5.90% for summarization workloads. This is an efficiency trade-off under specific hardware and latency targets, not evidence that every NPU is faster. The power analysis uses device ASIC power, which is not whole-system energy or total ownership cost.
12 · RISK-KV: select omissions by post-recovery quality
RISK-KV: Quality-Aware Selective KV Cache Reconstruction for Distributed LLM Inference
Yuekai, Wang · 2026-10-21
RISK-KV retains exact KV anchors on both sides of target layers. Offline ridge regression predicts intermediate hidden representations, then the original model projections reconstruct KV. Layer selection uses the NLL change caused by actual recovery rather than tensor similarity or sensitivity to random noise alone.
Reading the diagram: offline calibration produces predictors and a retention policy; migration retains anchors, reconstructs missing layers, and resumes decoding.
Across three roughly 0.5B–1.5B models on WikiText-2, with 512 historical tokens and 512 subsequent teacher-forced tokens, an independent assessment at a 0.005 quality budget reports historical-KV savings of 33.3%, 33.3%, and 21.2%. These are raw cache omissions: predictor and auxiliary storage offset net savings. Small-sample NLL results do not establish long-form generation quality or distributed-serving speedups.
13 · Koala: transfer ownership before moving state
Koala: Non-disruptive Reconfiguration for Long-running Stateful Dataflows
Huang, Lei et al. · 2026-10-20
Koala separates logical key ownership from physical state location. Reconfiguration first transfers Key Lookup Table entries so new tasks can process events; values migrate on first access, leaving cold state behind. Old tasks forward in-flight records, upstream markers confirm routing changes, and watermarks cover both upstream and temporary peer channels.
Reading the diagram: forwarding and ownership transfer precede upstream rerouting. Hot-key access triggers state fetches; temporary channels close after old records drain.
In scale-out experiments, state transfer averages 8.7 times less than DRRS across real-world queries. Some baselines never drain their backlog within the ten-minute window, so the roughly 23-fold recovery gap has a lower-bound interpretation. Benefits depend on locality, and the protocol assumes no failures during reconfiguration; failures still trigger checkpoint recovery. This is relevant to elasticity and state semantics in long-running streaming jobs.
14 · Approval monitors: shared evidence matters more than vote counts
Correlated Failures of AI Approval Monitors
Zanolini, Luca · 2026-10-21
Monitors sharing a model or poisoned evidence source can fail together. The paper uses fail-prone sets from distributed systems to describe approval rules, separating mistaken approval of bad actions from mistaken rejection of good ones. Safety and liveness depend on whether sets from the two directions can jointly cover every monitor.
In payment-approval experiments, a source-aware rule reduces rejection of correct payments from 7.4% for the best vote-counting rule to 0.9% at equal safety. The preregistered worst-case test nevertheless fails: one attack leaves all approvers reading a single source, defeating joint safety and liveness for the twelve language-model monitors. Adding a non-persuadable rule engine restores the measured condition. This supports source diversity, not guarantees against unseen attacks; it also assumes legitimate requests and fields checkable against external records.
15 · Context compaction: lost provenance can become false authority
Compaction Laundering: Provenance Destruction in Context-Compacting Coding Agents
Dabirsiaghi, Arshan · 2026-10-21
This security study identifies a specific boundary: flattening typed conversations into text can make role labels inside tool content indistinguishable from real user turns. A subsequent summary may promote external text into user authorization. Controlled comparisons report 87%–100% laundering across five flattening serializers and zero under the tested attacks across four role-typed approaches.
Reading the diagram: Follow the transition from typed messages to plain text: provenance is lost there, before the summary is carried forward as memory.
Most experiments exercise pinned summarization prompts and serialization components, rather than all nine products end to end. A separate paired real-pi-TUI experiment reports roughly 50% command issuance against zero in the clean arm. The design lesson is to preserve source metadata that documents cannot forge and verify authorization provenance after compaction. Zero observed success covers only the tested attacks, and provenance preservation does not replace runtime permissions.
16 · Independent verification: test evidence needs a fresh context
Who Verifies the Builder? Toward Independent Assurance for Agent-Produced Software
Sion, Sion Israel, April, Alain · 2026-10-16
The framework first freezes specification-derived tests before the verifier sees implementation, then strengthens evidence with implementation-aware techniques such as mutation testing. The builder receives counterexamples rather than hidden test code. Across 46 seeded faults in three stateful services, two-phase verification by the same model in a fresh context detects 95.4%, versus 85.1% for builder self-tests.
Reading the diagram: Specifications feed the verification contract and independent evidence; the builder receives counterexamples, and the assurance report informs a human release decision.
All final builder programs themselves pass the hidden tests: the observed problem is incomplete self-evidence, not widespread faulty generated implementations. A different-provider verifier does not improve detection. The verification harness also mishandles parameterized tests, creating many false failures. The useful lesson is to examine independence, test oracles, and release gates together; another verifier agent is not automatic quality assurance.
17 · AI-GREEMENT: connecting protocol knowledge, verification, and experiments
The End of Consensus Research?
Burgelin, Clément et al. · 2026-10-21
The authors build a consensus-specific harness combining a conceptual knowledge base, modular implementation skeleton, Lean safety verification, and performance/fault feedback. Its resulting A-GREEMENT protocol uses leases for local reads and selects write paths according to key popularity, topology, and read waiting costs.
Reading the diagram: The columns supply protocol knowledge, an execution skeleton, and Lean verification; research, development, and verification iterate with performance and fault feedback below.
On a CPU cluster with emulated network conditions, equally averaging six deployments with five or nine replicas, mean latency improves over the fastest baseline by 12.2% for 50%-write workloads and 33.3% for 5%-write workloads, while matching it on read-only workloads. Larger replica configurations are excluded from these averages. Formal verification currently covers safety, not liveness; the general-agent comparison lacks the same knowledge support, so its gap cannot be attributed entirely to the harness. The title is a question about changing expert-guided research practice, not evidence that consensus research has ended.