What matters in this issue
A coding-agent run that looks likely to fail is not automatically worth restarting: the original might still finish, while a replacement has only the remaining budget. Two studies examine this decision from complementary angles. One tests whether early traces distinguish attempts on the same task; the other asks whether a prediction supports an action with adequate statistical evidence.
Continue or restart: limits of early trajectory prediction
Continue or Restart? The Limits of Early Failure Prediction in Coding Agents
Srinath Srinivasan, Lunxiao Li, Tim Menzies, Wesley Assunção · 2026-10-26
After a coding agent has run for a while, a monitor might use tool calls, edits and tests to decide whether to continue. But a predictor may merely recognize task difficulty: it can separate easy issues from hard ones without identifying the failing attempt among multiple runs on the same issue. The latter distinction is what directly supports selective intervention on a particular run.
The authors analyze 147,002 traces from two public corpora spanning four model–framework settings. They test count features, initial-text embeddings, task-conditioned rankers and a fine-tuned Qwen3-0.6B monitor, reporting both pooled ranking and ranking of successful versus failed attempts within the same task. At 20k tokens, the neural monitor’s three-seed mean within-task AUC across 32,406 attempts is 0.50, with an interval of 0.48–0.51. That is close to chance in this setting; it does not prove that every possible monitoring signal is ineffective.
Reading the figure: the left panel shows within-task ranking at different landmarks; the right shows the fraction of runs still active. Later decisions cover fewer runs.
The authors replay three policies under shared token caps: continue, restart according to a score, or restart randomly at the same stopping rate. At a 75k cap, learned restarting improves resolution over matched random restarting by up to 1.98 percentage points. A decomposition attributes almost all of that gain to allocation across tasks—avoiding stops on easier issues—rather than selecting attempts within a task. Continuation achieves higher resolution at lower mean token cost than both restart policies in all four settings at 75k, and in 216 of the 224 budget–threshold configurations overall.
This negative result makes the case for evaluating prediction, intervention and budget together. It concerns cold-restart replay of recorded traces; preserving edits, richer signals or other workloads could change the outcome. The within-task evaluation and shared-budget policy comparison deserve particular attention alongside pooled AUC.
Beyond monitor accuracy: measuring restart decision value
Monitor Accuracy Isn’t the Answer: A Decision-Value Framework for Evaluating AI Agent Restart Monitors
Siddharth Jain, Venkat N Vedam · 2026-10-27
This paper takes the question one step further: an accurate monitor need not make restarting beneficial. The decision depends on both the original run’s chance of succeeding if continued and the replacement’s chance within the residual budget. In a mathematical control with an independent clean restart and zero monitoring overhead, the authors hold accuracy at 90% and vary only replacement success probability; the resulting change in success ranges from a five-percentage-point loss to a 35-point gain. This is an analytical example, not measured monitor performance.
The authors formulate whether a compressed history preserves the information needed to choose between continuation and restart, and construct an audit using simultaneous binomial intervals. It can abstain when evidence is insufficient rather than force a recommendation. The empirical study contains 194 externally evaluated executions on 19 selected SymPy issues from SWE-bench Verified; 130 are used in a retrospective audit. An apparent 2.29-point restart gain came from one eligible failed attempt; another model saw a 42.29-point loss. None of 103 comparisons with live data met the authors’ certification threshold.
Both papers place action value ahead of prediction metrics. This one contributes a decision identity and a small-sample audit with abstention, rather than a large-scale failure predictor. Its results are limited to the selected tasks and runtime configurations, and the certificate depends on assumptions including independence. A restart retaining edits is a different intervention and cannot simply inherit the clean-restart result. The paper has no suitable architecture figure for the central mechanism, so this account uses text.