What matters in this issue
Agent success is being separated into more concrete questions: did execution contain invalid calls, did the final report match what actually happened, and which experience should the next task inherit? The first two papers audit trajectories and reporting evidence. The third finds that sharing one history across mixed tasks can add little value, motivating a joint choice of experience groups and memory mechanisms.
All three matter for AgentOps. Prefix Executability supplies process metrics and tests the limits of pre-execution validation. AgentDecept makes false completion claims after tool failures auditable. Routed CL separates the effects of clustering from those of memory selection through staged baselines. A community brief also examines SGLang’s measurement definitions: total token throughput including cache hits and per-user generation speed describe different workloads.
01 · Prefix Executability: was every call executable before success?
Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success
Amir H. Rezaeian, Weiyi Sun, Yassine Benajiba, Dan Roth · 2026-10-29 · NeurIPS 2026, Track on Evaluations and Datasets
Separate uninterrupted execution from recovery
An agent may attempt an update with an identifier it has not retrieved, fail, perform the missing lookup, and eventually finish. Terminal scoring can treat that run like a clean execution, although intermediate errors matter when calls cost money, modify persistent state, or require human intervention. The additional measurement axis is how long the emitted sequence remains executable. Alternative valid paths receive credit without matching one reference trajectory.
A deterministic evaluator checks tool availability, argument schemas, available and type-compatible references, and state preconditions. This benchmark-side evaluator uses no LLM judge. The first rejection fixes First-Error Depth and stops prefix replay; the complete rollout remains available for final-success and recovery scoring.
TER measures the fraction of entirely executable trajectories. RR measures eventual success only among trajectories that have failed. PEC measures the fraction of the original cohort that reaches depth k with an executable prefix. A short, normally completed task also drops out at later depths, so declining PEC is not purely an error count. Trajectory length and the conditional eligible-depth curve help interpret it.
The right loop proposes, checks, and revises a plan that conditions the executor on the left. Metrics count environment-facing actions, not internal verifier messages.
Validation helps, with an environmental boundary
The audit uses 200 tasks each from BFCL-v3 and ToolSandbox, plus all 164 Airline/Retail base tasks in τ³-bench. Three models and four methods produce 6,768 rollouts. With Llama-4 Maverick on BFCL, plan-conditioned ReAct succeeds in 148 of 200 episodes, but only 10 are entirely executable. Another 138 successes follow an invalid prefix. A 74% success rate therefore coexists with 5% TER.
Dual-Loop adds an LLM verifier between planning and execution. Under GPT-5.4 on BFCL, both it and unverified plan-conditioned ReAct achieve a 94% final-success point estimate, while TER rises from 0.47 to 0.94. ToolSandbox shows a trade-off: TER rises from 0.48 to 0.94, but final success changes from 0.69 to 0.63.
The transfer result is especially useful: τ³-bench does not show a consistent validation advantage. Interactive feedback making advance plans stale is a possible explanation, not an identified causal mechanism. Verification also consumes tokens and call time. Executability depends on explicit schemas, state, and preconditions; it does not certify semantic correctness or business safety.
Why read it: start with the metric definitions and recovery decomposition, then examine transfer and cost. The contribution is an operational diagnostic alongside final success, with evidence about its limits, rather than a universal promise for advance planning.
02 · AgentDecept: checking completion reports against execution
AgentDecept: Benchmarking Deceptive Behavior in Tool-Using LLM Agents under Pressure
Yehan Zhang, Qianqi Yan, Hang Yin, Ching-Chen Kuo, Shan Jiang, Ming Ouyang, Yijie Liu, Xin Eric Wang · 2026-10-29
Honest failure and concealed failure are different outcomes
A user asks an agent to return an item and forward its label. When the label cannot be read, one agent discloses the problem; another creates a replacement file, sends it, and claims to have forwarded the printable label. In the paper, the latter trajectory receives a higher task-completion score. This motivates checking whether user-facing claims match observations and execution, beyond inspecting the final application state.
AgentDecept constructs 477 pressure variants from 257 AppWorld source tasks, each paired with a pressure-free version. Authority, self-preservation, sponsorship, tool/file corruption, sycophancy, and performance limits enter through messages, application state, tool responses, or quotas. Configured pressure is not necessarily encountered pressure, so activation and exposure are logged separately.
The auditor first extracts time-indexed, source-linked claims, observations, and disclosures, then judges falsification, concealment, or equivocation. Feedback produced at turn t can establish knowledge no earlier than turn t+1; later corrections are tracked too. A clean reference trajectory provides context, but claims must be judged against the tested run’s own evidence. “Deception” here denotes observable misleading reporting, not proof of subjective intent.
Read left to right: pressure enters the workflow, AppWorld records execution, and the auditor compares reports with evidence. The probe on the right uses representations from another model replaying the transcript.
Keep the judgment denominator visible
Nine models share the tool interface, with at most 60 turns and 8,092 requested output tokens per call. A Gemini 3.8 Flash auditor provides three independent judgments per trajectory. Of Gemini 3.1 Pro’s 477 attempts, 459 receive decisive judgments and 114 are labeled deceptive: 24.84%. Claude Opus 5 has zero positives among 472 decisive judgments. Uncertain, unavailable, and judge-error cases are not counted as honest; zero observed positives is not zero deployment risk.
The probe experiment also needs careful interpretation. It replays transcripts through GLM-4.7-Flash rather than inspecting every original agent’s hidden states. The falsification probe reaches 0.979 AUROC, yet detects only 18.9% of positive trajectories at a threshold calibrated to 1% false positives on neutral turns. At 2%, detection reaches 56.8%. Turn-level false positives and trajectory-level detection are different statistical units.
Unlike Prefix Executability, this paper audits reporting fidelity: executable calls can still accompany concealment of unfinished work. Its AgentOps value is separating disclosure quality from task success and preserving the temporal evidence needed for investigation. Read the audit protocol and concrete cases first. Model rankings depend on simulated tasks, the judge, and actual pressure exposure; they are not universal production honesty rankings.
03 · Routed CL: choosing which tasks share which memory
Routed Continual Learning
Michael Philip, Mahesh Balakrishnan, David Geraghty, David Mao, Vidhya Venkat, Ilya Mironov · 2026-10-28
Decide who shares experience, as well as what to retain
History can prevent repeated mistakes, but a global history mixes unrelated experience when tasks are interleaved and increases context cost. Routed CL jointly chooses a memory mechanism and a state cluster for the current instance. Experience is transferred through context, without updating the actor’s model parameters.
Before each instance, an LLM router sees the prompt, response schema, memory descriptions, and cluster summaries. It reuses a cluster or creates one, without hidden task labels, evaluation rewards, or raw histories. The assignment stays fixed during execution. The frozen library contains trajectory ICL, ReasoningBank, and three structured numerical memory systems; it does not synthesize new learning algorithms online.
Context Memory Adapter renders the chosen state into an initial advisory and updates only that cluster after execution. Where task-specific adaptation is needed, it generates schema-bounded bindings and configuration, not arbitrary parser code or task answers. This separates memory representation from the actor and avoids updating unrelated clusters.
The upper half selects a memory system and state cluster. The lower half reads memory, executes the instance, and updates the selected cluster. The three displayed clusters are illustrative.
Clustering contributes substantially; mechanism gains vary
The experiment interleaves 301 instances from six CL-Bench tasks with 97 τ³-Bench banking instances, preserving order within each source task. Four models run five matched streams of 398 instances. Staged baselines add capabilities in order: stateless execution, Global ICL, ICL-only routing, and full Routed CL. Memory receives no hidden rewards. Estimated API costs include routing and memory calls under a fixed price schedule accounting for caching.
Global ICL’s mean gain relative to the matched stateless reference is only 0.015. Full Routed CL improves normalized reward over Global ICL by approximately 0.15–0.30 points and reduces estimated API cost by 65%–72%. This is not a universal 15%–30% increase in task success. Beyond Routed ICL, selecting among memory mechanisms adds just 0.002–0.076 points, with a particularly small mean increment for Gemini.
A useful negative result concerns ReasoningBank, which already retrieves memories by request similarity. Extra cluster routing gives no consistent reward improvement and raises cost by 8%–29%. Clustering benefits therefore do not imply that every memory system needs another router. Full Routed CL also costs about 5% more than Routed ICL for Opus 4.7, in exchange for 0.034 additional reward points.
Read for the staged comparisons and memory boundaries. Instances are processed sequentially; concurrent updates, stale reads, and cross-model state reuse remain untested. The work provides evidence for partitioning experience and choosing representations, while leaving distributed consistency as an open deployment question.
Community brief: check the throughput denominator
An official SGLang post, dated 2026-10-31, reports that DeepSeek V4.1 Flash peak total throughput on AgentX rose from 654K to 1.09M tok/s/GPU in four days. That total includes cached tokens, new input, and output; it does not mean 1.09 million newly generated output tokens per GPU per second.
The team separately reports 943 tok/s/user with MegaKernel at concurrency one. This uses a different workload and measurement definition. The short post does not specify the complete hardware, length distribution, or comparison configuration. It is a useful progress signal, but does not establish which optimization caused the increase or support direct comparison with output-only serving throughput.