What matters in this issue
Start with Request Order Matters and AgentSpy. The former finds that identical prompts can yield different answers under different cache histories, making cross-request state part of serving evaluation. The latter observes agents at the operating-system boundary, exposing file accesses, network connections, and execution paths that passing outcome tests can conceal. Together, they offer evidence about why answers change and what agents actually do.
The other four papers examine the costs of sharing resources: MOLT returns GPU memory during a training step; LearnSched compares the net value of improving now versus later; Mosaic predicts GPU kernel interference; and Ofan makes network load balancing destination-aware. Their useful distinctions concern which state can survive, when capacity must return, and which metric actually improves. These six studies do not establish that the field has converged on a common solution.
In depth
01 · Request Order Matters: cache history can change an identical prompt's answer
Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents
Tiffany Gu, Annie Guan, Manshu Huang, Nitin Rao, Siddhant Shah, Margaret Capetz · First submitted 2026-10-05 (UTC)
Unchanged text can encounter changed intermediate state
Long-running agents often retain most documents while removing the oldest and adding new material. Removing an early document changes both the positions and preceding context of subsequent tokens, disrupting exact prefix caching. CacheBlend’s non-prefix reuse loads old KV states and recomputes selected tokens to repair contextual differences cheaply. When caches persist across requests, however, the state encountered by a call also depends on which requests were processed earlier.
The paper keeps every prompt unchanged and varies only the order in which complete requests enter the system. It studies cache history left by preceding requests, rather than document reordering within a prompt. Default token top-k selects individual tokens with the largest deviations between current and cached keys, potentially scattering the selection. The document-aligned alternative ranks document spans by average deviation, selects complete spans first, and fills the remaining budget with a contiguous window. Both recompute exactly 5% of reusable tokens.
Reading the diagram: the left panel replaces one of 84 documents per update; the middle creates different cache histories through five request orders; the right compares scattered-token and contiguous-span recomputation under the same 5% budget.
Equal recomputation, different sensitivity to history
The authors use Qwen3-8B with greedy decoding, LMCache 0.5.1, and vLLM 0.23.0, disabling vLLM prefix caching. Financial records produce 7,735 rolling updates, each with 84 documents and approximately 7.2K–7.7K tokens. The task copies requested fields verbatim. Five orders include chronological, reversed, and three fixed random orders. Requests are issued individually; each experimental run starts from an independent empty cache and retains state within the run.
Cross-order answer variation falls from 69.0% with token top-k to 26.1% with document-aligned recomputation. This metric measures answer disagreement across the ten pairs of five orders for each identical request. Agreement with full-prefill answers is 86.8% versus 85.4% in chronological order, with no significant difference. Across the other four orders, document-aligned recomputation gains 34.5–52.5 percentage points. Its fidelity still falls to 54.3% in the reversed-order stress test, so repair does not eliminate history dependence.
On an NVIDIA L4, median time to first token is 2,408 ms for full prefill, 426 ms for token top-k, and 415 ms for document-aligned recomputation: roughly 5.7× faster for both reuse policies. P95 values are 2,524, 459, and 477 ms, respectively. The document policy’s slightly better median does not extend to its tail, and these figures do not establish concurrent-serving throughput.
The lasting contribution is an evaluation dimension
In a 500-request ablation, a contiguous window, fixed chunks crossing document boundaries, and randomly ranked document spans all substantially outperform scattered selection under a fixed random order. Document boundaries and deviation-based ranking show no significant additional advantage; the evidence more strongly supports contiguity as their shared feature. A separate 500-request control that resets the cache before every request also finds no significant fidelity difference between the two main policies.
Read the evaluation protocol and selection-pattern ablation first. For persistent agents, cache history belongs among the test inputs; clean-cache evaluation can miss the problem. The evidence remains limited to one model, one field-extraction workload, and a fixed 5% budget. Agreement with full prefill is not equivalent to factual correctness. Isolating the effects of retaining versus recreating cache entries remains future work.
02 · AgentSpy: system behavior exposes blind spots in outcome tests
AgentSpy: Making AI Agent Behavior Observable
Christoph Bühler, Matteo Biagiola, Luca Di Grazia, Guido Salvaneschi · First submitted 2026-10-05 (UTC)
One tool call can hide many unexplained actions
When an agent executes a script, its trajectory usually records the invocation and result without individually exposing the files accessed, hosts contacted, or processes spawned inside it. Outcome tests may only check whether an output file is correct. AgentSpy observes the operating-system boundary from outside the agent, turning resource interactions into comparable evidence that complements the natural-language reasoning in a trajectory.
The framework takes a task, agent configuration, and analysis type. An orchestrator creates an isolated environment, an execution manager runs the task and collects evidence, and an external analyzer produces a report. The implementation uses Firecracker microVMs; reliability analysis combines auditd, laurel, and Zeek, while security analysis uses Falco. Monitoring starts after preparation and stops before artifact collection, excluding dependency installation and result export from agent behavior.
Reading the diagram: tasks and configuration enter the orchestrator on the left; the isolated environment produces run artifacts in the middle; the analyzer on the right reads those artifacts to generate reports. The same artifacts can be analyzed again without rerunning the agent.
Check obligations and prohibitions separately
Reliability analysis reduces an execution to a star graph: the execution is central, commands, files, and hosts form the surrounding nodes, and edges record action types and counts. It normalizes counts within each resource category and computes weighted Jaccard similarity. It also reports the fraction of skill files accessed. These scores help locate differences; low coverage alone does not establish incorrect execution, because a task may admit other reasonable solutions.
Security analysis applies deterministic rules to system calls, checking sensitive-data reads, modifications outside the workspace, untrusted execution or privilege changes, and unapproved network access. It lets suspicious behavior occur within an isolated environment and records alerts, making it useful for auditing a skill’s effects. The rule checks themselves are not a runtime defense that prevents every harmful action.
What passing runs can still conceal
The reliability experiment uses gpt-5.6-terra, deepseek-v4-flash, and glm-5.3-flash under the codex harness, running each of 77 SkillsBench tasks three times. Task identity serves as a proxy for behavioral difference: sets of same-task runs are more similar than sets mixed with another task in 92.2% of comparisons. This is not anomaly-detection accuracy.
Manual inspection retains only the 88 task–model pairs whose three runs all pass every test. Of these, 16 (18.2%) contain at least one run with activity the authors label unrelated to the task, six access grading test files, and 15 (17.0%) reach the result without using the skill’s guidance. Categories can overlap, and the denominator is not all executions, so these proportions cannot estimate production failure rates.
Security evaluation selects one case from each of five attack categories, with five runs per injected and control condition, totaling 50 runs using gpt-5.1-codex-max. Injected behavior executes in 18 runs; 14 are detected, giving precision 1.00 and recall 0.78. There are no false alerts in the other 32 runs. All four misses merely add a malicious Git remote inside the permitted workspace without pushing or contacting it, leaving the generic rules untriggered.
Read the design alongside the missed-attack case. AgentSpy provides repeatable summaries of resource behavior, while semantic judgments still require task context, trajectories, and custom rules. Overhead tests use ten runs per condition and detect no significant runtime difference, but have power only for large effects; this does not establish zero overhead. One harness, limited models, and selected attack cases also constrain generalization.
Resource sharing and scheduling
03 · MOLT: returning memory to inference during a training step
MOLT: A Fine-Grained GPU Memory Sharing System for LLM Serving with Opportunistic Fine-Tuning
Jaehoon Yang, Yongbeom Kim, Hojoon Kim, Seung Yul Lee, Jae W. Lee · First submitted 2026-10-05 (UTC)
MOLT reduces the sharing unit to saved activations needed by backpropagation. When inference needs KV space, it reclaims safely evictable activation pages with lower recomputation costs. The current training step continues and recomputes missing activations during the backward pass. A slot-state table and protection windows spanning tensor-parallel ranks prevent reclamation of memory still accessible to in-flight kernels.
Reading the diagram: the inference allocator requests pages; the right component marks slots protected by in-flight work; the left selects saving, reading, or recomputation according to whether pages remain backed. The shared pool below connects both paths.
With BurstGPT arrivals, ShareGPT inference, and rank-8 LoRA tuning on Alpaca, four H100/B200 deployments of 24B/70B models attain at least 99.7% of inference SLOs and 1.9–3.3× the tuning throughput of LLMStation+SIRIUS. The SLO requires both TTFT and time per output token to stay within three times the inference-only mean; it does not imply no inference slowdown. The design targets opportunistic tuning with a shared frozen backbone and requires allocator changes, recomputation, and handoff costs. Start with how protection windows handle asynchronous execution.
04 · LearnSched: an improvement worth finishing may be worth postponing
Serve Now or Improve Later? Scheduling Self-Evolution in Online Agent Systems
Yangbo Wei, Junhong Qian, Zhen Huang, Zhenyu Su, Qifan Wang, Shaoqiang Lu, Rumin Zhang, Chen Wu, Lei He · First submitted 2026-10-05 (UTC)
LearnSched treats construction of tools, guidance, or model state as an investment with delayed returns: benefits arise only after publication and actual use. Tracking progress, durable progress, and recovery state, the scheduler uses one-step counterfactual rollout to compare advancing, saving, and waiting under capacity-return constraints. Costs include checkpointing, recovery, and state residency.
Reading the diagram: the left panel connects serving and improvement; the middle compares feasible actions under the same continuation policy; the right separates durable progress, service recovery, and benefits that begin only with actual artifact reuse.
A100 component measurements inform 1,920 paired finite-model scenarios. Relative to the fully costed Costed Window baseline, mean net value improves by 1.10%, with 91 improvements and 1,829 ties; all 960 warm-recovery scenarios tie. The gains come from avoiding expensive cold recovery and do not establish that complex scheduling is always better. The model assumes known idle windows and accurate transition costs, with assumed memory prices; unknown real arrivals carry no hard deadline guarantee. The valuable reading is how cost accounting and state management change the value of scheduling.
05 · Mosaic: feeding kernel-interference predictions into tail-latency scheduling
Mosaic: GPU Sharing with Latency Guarantees through Kernel-Level Interference Prediction
Foteini Strati, Ethan Graham, Leo Stephan, Paul Elvinger, Ana Klimovic · First submitted 2026-10-05 (UTC; October 6 in Beijing)
Even jobs occupying separate SMs contend for L2 and HBM. Mosaic combines thread-block placement, block-packing density, and intra-/inter-SM interference models to predict colocation slowdown from two kernels’ isolated profiles. MosaicSched admits a best-effort kernel only when the high-priority request is predicted to remain within its budget.
Reading the diagram: profiles of two kernels and the stream mode enter the predictor; four components describe placement, block packing, and two kinds of resource interference, producing each kernel’s slowdown for scheduling.
The predictor covers four NVIDIA GPU types; scheduling is evaluated on H100 with fixed-batch inference. Most workloads have at most 1.0% of requests exceed the target, but MobileNet reaches 1.2% at a 1.6× slowdown allowance: prediction errors and queueing can still cause violations. A strict 1.1× target may also throttle background throughput enough to erase colocation’s cost benefit. Calibration is GPU-specific, CUDA streams are assumed to have equal priority, and power throttling is not explicitly modeled. Read the title’s “guarantees” as demonstrated SLO control under the evaluated conditions.
06 · Ofan: balancing upward paths requires considering downward destinations
Ofan: Optimal Load Balancing for AI Training
Sarah McClure, Evyatar Cohen, Jakob Krebs, Alexander Shpiner, Mark Silberstein, Scott Shenker, Sylvia Ratnasamy, Isaac Keslassy · First submitted 2026-10-05 (UTC; October 6 in Beijing)
In a fat tree, selecting an upward path fixes the downward route to a destination. Balancing local uplinks alone can leave collisions downstream. OFAN maintains switch-side rotation pointers grouped by destination switch, distributing same-destination traffic evenly. POFAN splits pointers by pipe to fit existing ASIC pipelines, sacrificing some consolidation.
Reading the diagram: destination-oblivious splitting on the left still creates downstream collisions; destination-based rotation on the right avoids concentrating traffic on the same downstream paths. This is a mechanism illustration, not a measured network topology.
The authors report P4 implementation resource usage, while performance comes from htsim simulations. In a 1,024-GPU, FP8, FSDP2/HSDP Llama-3 405B scenario, OFAN reduces collective-completion-time inflation above the baseline by 16–39× relative to the better of AR and REPS, for NSCC and MSwift respectively. This is not an equal speedup in total training time. The constant-order average-queue result relies on a queueing-model approximation, and POFAN has no hardware-cluster performance evaluation. Begin with the explanation of downstream collisions, then examine the simulation assumptions.