What matters in this issue
Two problems can disappear behind aggregate agent scores: whether the measurement is trustworthy, and whether resources follow demand during multi-turn execution. An oracle audit shows how a scoring function’s side effects can make a later evaluator read the wrong terminal state. Prorsa uses lifecycle signals, including tool returns and sandbox startup, to anticipate prefill and decode capacity needs.
Start with these two papers. The first supplies concrete tests for evaluation infrastructure; the second turns the structure of agent execution into scheduling information. Three further papers examine resource migration for generative-reward training, knowledge-driven Kubernetes tuning, and responsibility attribution in workflow failures. Josh Rosen’s tracing article adds a practical perspective: recording what a model said is insufficient to explain what changed in the surrounding environment.
01 · Oracle auditing: high scores can come from an overwritten terminal state
When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
Junjie Chen · arXiv v1 · 2026-10-12
Test the instrument before interpreting the score
Agent benchmarks connect tasks, simulated users, tool environments, and reward functions to produce a success rate. Junjie Chen treats the reward function itself as software under test: if grading changes the database, does the final environment still represent the agent’s actions? This question precedes adding another capability task because faulty grading can contaminate comparisons across models.
The defect occurs during reference-solution replay. The reward function resets the database and executes reference actions to construct the expected state, without restoring the previous state. Its immediate return value can be correct, and the benchmark’s own runner consumes that value without suffering this downstream error. The author’s harness, however, scores separately after the session ends. Its second read sees the reference solution’s database. An episode that performs no writes and merely says goodbye can consequently receive a database pass. The failure depends on the combination of the scoring side effect and its caller.
Check state integrity and known answers separately
The auditing interface requires a fresh environment, a comparable state signature, an oracle call, and identification of the class into which defects can be injected. Five invariants check state preservation, repeat-call consistency, scoring-order independence, restoration on exceptions, and terminal-state preservation. Three known-answer controls reject empty behavior, accept the reference solution, and accept permitted output variants. A state-clean oracle that always returns success escapes state checks alone, so the two families serve complementary purposes.
Read left to right: expose four oracle operations, run invariants and known-answer controls, then establish whether injected behavior is distinguishable before computing detection results.
Seven defect classes become 21 concrete variants, tested in retail and airline domains sharing one implementation and in a synthetic domain: 63 combinations altogether. Differential probes distinguish 57 from the repaired implementation; the suite detects 54. Six remain unresolved. The three misses arise from the same exception-swallowing variant across the targets. This supports an auditing method with explicit blind spots; coverage of seven classes does not mean every instance is detected.
Corrected scores and portability limits
For 4,320 archived episodes, replaying recorded tool calls against clean databases reduces database passes from 4,194 to 2,883 and overall success from 94.3% to 65.5%. Database pass and overall success have different definitions: the latter also requires textual conditions. All 33,402 tool observations match the archived strings exactly. The 137 step-limit episodes that bypassed the defective path also retain their original labels, providing cross-checks for the correction.
A separate component experiment over 3,240 episodes shows that the language of policy documents, tool descriptions, and error strings can all change the measurement. A second task domain even reverses the user-language effect. These results do not establish universal superiority for either language, and cross-batch model drift remains possible. A more direct portability limit appears in a second independently implemented benchmark: its API log cannot reconstruct the reference terminal state, so the study estimates no detection rate there.
Prioritize the method and minimal defect reproduction. The transferable contribution is to validate terminal-state integrity, repeat scoring, and known answers in the evaluator itself. This requires a resettable, replayable environment, beyond a saved score table.
02 · Prorsa: allocating prefill and decode resources from trajectory lifecycles
Prorsa: Unleashing the Potential of Prefill-Decode Disaggregation in Agentic Reinforcement Learning
Chiheng Lou, Xiangwei Deng, Bangjun Xiao, Xiaodong Ji, Yikai Zhao, Xin Jin, Hailin Zhang · arXiv v1 · 2026-10-12
Multi-turn rollouts change the rhythm of computation
Single-turn RL often prefills a batch at the beginning and then spends a long interval decoding. Tool-using agents instead cycle through generation, tool execution, and new model requests. Tool returns continuously introduce prefills, while asynchronous sampling adds trajectories. Under prefill-decode colocation, a long prefill in one attention data-parallel group can stall decode work in others. Separating the stages helps, but a fixed GPU ratio still cannot follow changing demand during training.
Prorsa uses trajectory lifecycles visible to the RL framework to predict near-term demand and change worker roles. It chooses parallelism configurations separately for the two stages and includes decode KV capacity in allocation. A token-rate-only allocation could otherwise lack space for concurrent long trajectories.
Trajectories below cycle through prefill, decode, and tools. Above them, lifecycle signals feed demand predictions and changes to the number of workers in each role.
Predicting work that will arrive next
A sample starts a prewarmed or cold sandbox and places its first request in the ready queue. Prefill completion admits it to decode; a completed tool call creates another request, while a finished trajectory enters reward evaluation. The predictor combines upcoming samples, starting sandboxes, queued requests, and tools in progress. Historical durations and elapsed time estimate arrivals within the prediction window.
Queued and in-flight prefills contribute their full prompt lengths. Requests expected to complete prefill contribute anticipated decode KV requirements. The predictor does not forecast subsequent output lengths. Runtime observations update duration statistics and bias correction. The role switcher combines these estimates with existing decode KV occupancy, measured prefill processing capacity, and the GPU budget. Experiments use a ten-second prediction window, making tool waiting time an advance signal for capacity planning.
Prediction is imperfect. For MiMo-V2.5-Pro on the two agent tasks, weighted absolute percentage error is 5.40%–14.37% for prefill and 5.19%–13.69% for decode admissions. General-agent tasks have higher errors because tool durations vary more. This suggests that benefit depends on online statistics keeping up with workload changes.
Separating the sources of speedup
The implementation builds on Verl and SGLang. Experiments use 16 servers with eight GPUs and eight 400 Gbps RDMA NICs per server. Models are FP8 Qwen3-235B-A22B, DeepSeek-V3.2, and MiMo-V2.5-Pro. Agent workloads use SWE-smith and a general-agent mix of APEX-Agents, Toolathlon, and MCP-Atlas, with GRPO group size 32 for agent tasks.
Across synchronous execution and asynchronous settings with maximum staleness 2 and 8, the two agent tasks achieve 1.11–4.05× rollout speedup over PD colocation. This combines stage separation, parallelism choices, and dynamic scheduling; it cannot all be credited to prediction. Against tuned static PD ratios for MiMo-V2.5-Pro, role switching contributes 1.0–1.44×. The paper also reports trajectory-completion latency improvements, a different metric from the duration of a rollout step.
The single-turn AIME25 math task exposes the boundary. Normalized rollout time relative to colocation ranges from 0.86 to 1.28, with slowdowns in most settings. Prefill and decode overlap less, so transfer and switching overheads are harder to amortize. Read the lifecycle predictor and static-ratio ablation first. The work makes waiting structure in long trajectories useful for scheduling, but benefit depends on stage overlap, KV pressure, and tool-duration stability. It does not establish a speedup for all RL training.
03 · PivotRL: moving GPUs between rollout, rewards, and training
Efficient Resource-Shiftable RL Post-training with Generative Rewards
Yanfeng Zhao, Haoyang Li, Sheng Lin, Yuming Zhou, Jiawei Jiang, Bin Cui, Fangcheng Fu · arXiv v1 · 2026-10-12
Generative reward models produce long sequences themselves, allowing the bottleneck to move among rollout, reward evaluation, and policy training. PivotRL pools cluster CPU memory into a global parameter store so GPUs can fetch the model needed by their new role. Shared waiting queues support sequence migration, while a planner compares sustained throughput gains with switching costs.
The monitor, planner, and executor form a control loop; global parameter storage and stage schedulers support the actual migrations.
Across four model groups on 128 H20 96GB GPUs, using DAPO-Math-17k for training and AIME-24 for validation, end-to-end throughput is 1.81–2.05× that of colocation and 1.50–1.95× that of static disaggregation. Experiments repeat three times. Role changes take seconds, sequence migration requires re-prefill, and an overly long planning horizon encourages excessive switching. The useful contribution is incorporating migration cost into the decision. Similar convergence curves in this math-reward setting do not establish the same result for long-running tool interactions.
04 · Kubernetes tuning: turning expert rules into configuration constraints
Domain-Knowledge Driven Configuration Tuning of Kubernetes Workloads
Federico Imberti, Luca Cavagnis, Stefano Cereda, Giovanni Gibilisco, Claudio Menghi · arXiv v1 · 2026-10-12
This industrial study organizes expert knowledge into configuration rules, then uses either a heuristic or an optimization solver to produce CPU and memory settings. The catalog contains 65 rules; the evaluation focuses on its 36 workload-level rules. The objective combines lower cost with fewer constraint violations, avoiding resource reduction based on utilization alone.
The catalog defines the configuration problem, and two solver paths produce values. The diagram contains no online trial-and-error control loop.
The evaluation covers 14,742 workloads from two industrial applications across eight clusters. On an Apple M4 Max with 36GB memory, the solvers average 0.71 and 45.54 seconds per extraction, respectively, illustrating the trade-off between speed and more extensive constraint solving. Costs are estimates based on public CPU and memory prices, rather than realized bill savings. The reported mean costs and savings differences also contain arithmetic inconsistencies, so the exact savings should not be adopted directly. Prioritize the rule modeling and solving process. Constraint satisfaction cannot substitute for online performance validation.
05 · Workflow failure taxonomy: responsibility behind a shared symptom
Why Agentic Workflows Fail: A Taxonomy Derived from Literature and Empirical Data
Xin Li, Rami Bahsoon, Tao Chen · arXiv v1 · 2026-10-12
The taxonomy records responsibility source, concrete failure type, workflow stage, and agent capability, separating a broken tool service from an agent choosing the wrong tool. It combines 397 literature records with 1,196 GitHub failure episodes from 229 repositories into a dictionary of 65 types.
The upper path constructs and analyzes the primary dataset; the lower path tests coverage on later literature without mixing that hold-out into the primary distributions.
In the valid stage–capability matrix, iterative execution paired with actuation accounts for 771/1,189 GitHub records (64.8%), versus 56/397 literature records (14.1%). Repository-level resampling and other checks preserve the contrast, but it describes differences in what the sources reveal, rather than production failure rates. Of 69 later-literature failures, 68 map to existing types. Read it as a diagnostic classification framework. The study has not established improved repair success from using the taxonomy, and human coding assisted by LLMs can still miss context.
Further reading: connect traces to environmental evidence
Agent Traces: Patterns for Capturing Agent Execution · Josh Rosen · 2026-10-12
Rosen surveys tracing through model interfaces, harness instrumentation, native session histories, distributed tracing, environmental evidence, semantic evaluation, and test datasets. The most useful distinction is that requesting a tool call establishes neither that it ran nor that the intended state changed. File diffs, test results, page snapshots, and database changes should be linked as artifacts to the operations that produced them.
This connects to the oracle audit: a call trace alone may not reconstruct a trustworthy terminal state. The article also explains why production traces can become regression cases without being inherently replayable, given environmental changes and nondeterministic generation. It is an architectural survey, with no new failure-detection benchmark or product performance comparison. The original post supplies no standalone architecture figure suitable for this mechanism discussion.