Original: Use NVIDIA NeMo to Turn Agent Failures into Tests and Harness Fixes
Patrick Moorhead, Sean Lopp, Moon Chung · 2026-11-04
Start with the action that violated the task
An assistant can draft a perfectly reasonable project update and still fail by sending it without confirmation. NVIDIA’s tutorial uses this case to show why reliability improvements may belong in prompts, tool descriptions, routing, memory, or execution code rather than in model weights. The useful unit is a trace-supported failure that can become a testable repair target.
The example is a Hermes enterprise assistant connected through MCP to a fictional company’s chat, email, files, and tickets, running inside an OpenShell sandbox. A request to draft an update led to a send_message call. Diagnosing the failure requires the user request, applicable behavior requirements, and executed action together; the final wording alone is insufficient.
From traces to an issue with evidence
NeMo Compass analyzes stored traces and evaluation evidence. The tutorial supplies 42 JSONL traces and ETHOS.md, which describes expected behavior. Compass returns structured issues, supporting examples, improvement directions, and trace IDs linking findings to individual runs.
Those links make the issue actionable: the developer can connect a preparation request to a sending action and the rule requiring confirmation. The same evidence can seed a regression test. The article also mentions internal advantages on larger trace collections, but supplies no public comparative accuracy measurement. The strongest evidence here is the explicit workflow, not a general detection-performance claim.
Turn the requirement into a verifier
NeMo Eval Author supplies coding-agent skills and workflows for Harbor or NeMo Gym evaluations. Inputs include Compass findings, raw traces, ETHOS.md, and the agent repository. Developers still need to inspect generated evaluations for fidelity to their requirements.
The example creates four Harbor tasks similar to drafting a response. The verifier checks for an unconfirmed sending-tool call rather than merely judging final-answer quality. Requirements and tool definitions can also seed tests before production traces exist; subsequent failures can expand coverage.
The demonstrated change is confined to Hermes’s SOUL.md. It distinguishes preparing a message from sending it and explains that a technical token returned by a preparation tool does not constitute consent. This is a prompt-level intervention, not evidence of an unbypassable authorization boundary.
What the small evaluation establishes
| Evaluation set | Baseline passes/runs | Updated passes/runs | Tool calls |
|---|---|---|---|
| Development: 2 tasks, 3 runs each | 3/6 | 6/6 | 10 → 6 |
| Held out: 2 tasks, 3 runs each | 4/6 | 6/6 | 28 → 15 |
The two held-out tasks were hidden from the coding agents proposing repairs. A new Compass pass no longer reported the sending issue, but found responses insufficiently grounded in retrieved evidence. That illustrates iterative diagnosis rather than comprehensive reliability: each set contains only six runs focused on one failure mode. It cannot estimate production prevalence or establish correctness for every sending scenario. Keeping held-out cases hidden remains important across later iterations.
A separate engineering example
In Dassault Systèmes’ LEO car-door-design workflow, QueryModel arguments appeared outside the expected options field, and an empty error message hindered recovery. Added validation reduced reported malformed calls from 142 to zero, with accuracy unchanged and no observed regressions in the other scenarios. An approximately 12-hour optimization cycle also reported a 28% relative improvement in the overall evaluation score and 35% fewer tool calls per run across six scenarios. These figures belong to the engineering example, not the four-task tutorial.
The transferable design connects four artifacts: traces establish what happened, requirements define the violation, verifiers make it repeatably checkable, and held-out tasks test whether a local repair transfers. That connection gives AgentOps teams a more maintainable improvement process than optimizing a single aggregate score.