This issue
Does naming a suspicious step after an agent failure identify an effective repair? ReplayRCA measures the effect of replacing one action with controlled replay and a placebo, separating post-hoc diagnosis from live oversight. AIFaultBench packages real AI software faults, triggers and reproduction records, providing a different kind of evidence: whether an original failure appears under documented conditions.
01 · ReplayRCA: Counterfactual replay for diagnosis and oversight
ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay
Jeremy Canale, Fabien Soulis · 2026-10-29
From agreement with labels to intervention outcomes
A long failed task can have multiple effective repair points. A benchmark that compares a selected step with one annotation can mark another successful intervention wrong. ReplayRCA does not claim to invent counterfactual replay; it makes replay the evaluation protocol. A diagnoser chooses a step and replacement tool action, the system restores the prior state, and the original worker continues until a final-state checker judges success.
Reading the diagram: both branches restore the state before the same action. The left executes a replacement; the right replays the original. The original worker continues in both, and their success-rate difference measures intervention benefit.
The control matters because a stochastic continuation may recover on its own, and restarting can also change the outcome. The authors sample five continuations per branch and subtract the original-action success rate from the replacement-action success rate, calling the difference causal repair. OpsWorld is deterministic: the system re-executes recorded actions from the initial state and checks observations byte for byte before intervening. This is not an experiment that rolls back an irreversible production service.
Explaining and preventing failure differ
The post-hoc diagnoser sees the entire failed trajectory and knows it failed; it can replace any one action. The live supervisor sees only the trajectory so far and decides before irreversible actions such as refunds or cancellations. It does not know the final outcome, may intervene once, and pays a five-percentage-point interruption cost in the score. Successful runs are included, so unnecessary intervention has consequences.
On 62 calibrated systematic failures, the highest post-hoc causal repair among 13 models was +51.3 percentage points. None of the 78 pairwise differences survived multiple-comparison correction, so the ordering is not a reliable ranking. Forcing a strong model to intervene at the last step cost 17.1 points, showing why intervention location matters. Live oversight covered 87 runs; a supervisor that performed well post hoc was weakest live, with a 14.5-point gap on the same runs.
Reading judgment: a useful evaluation protocol. It measures whether one replacement changes the outcome, not necessarily the earliest root cause, and does not show that multi-step repairs cannot work. The tasks use a synthetic, verifiably restorable back-office environment; external side effects, complex production services and longer time horizons require further study. Limited samples and differing model reasoning settings also constrain model comparisons.
02 · AIFaultBench: Packaging real AI software faults and reproduction evidence
AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems
Mehil B Shah, Mohammad Masudur Rahman, Foutse Khomh · 2026-10-30
The dataset contains 770 faults from 105 repositories across six system classes, including deep learning, LLMs and agents. Each record connects an original report and revision to environment setup, a trigger and execution evidence. The authors allowed up to three hours per fault and required all three executions to meet matching criteria. They report 652 reproduced and 118 not reproduced; obstacles include unavailable hardware or data, version drift and nondeterminism.
It complements ReplayRCA: AIFaultBench provides real software defects and reproduction materials, while ReplayRCA evaluates interventions in a restorable agent environment. Its sample favors active, popular public Python repositories; the reproduction rate is not the prevalence of faults across AI systems. Individual cases can still require external data, hardware or credentials. The paper has no suitable architecture figure to explain the dataset mechanism.