The benchmark combines tools, diagnosis and repair
Mercor describes APEX-SWE Observability as diagnosing and remediating production-style software failures using Terminus-2. The page says domain scores average 100 held-out prompts and illustrative tasks do not represent overall performance. Effort settings matter: the same model appears with different results at different effort levels.
The sample includes source code, Loki/Grafana observations and project/team tools and requires demonstrated observability-tool use. Its score therefore combines evidence acquisition, task understanding and repair rather than measuring root-cause judgment alone.
The sample connects symptoms to stateful requirements
The public task concerns transaction-pool balance checks that omit operator fees. Underfunded transactions enter the pool and fail later during execution. The requested repair accounts for base transaction cost, L1 rollup costs and operator fees, with different rules before and after Isthmus activation.
It must also update costs on transaction replacement, recheck pending transactions after fee changes and accept transactions again after sufficient balance is restored. Named helper functions, interfaces and test configurations constrain the implementation. Identifying the missing fee is only the beginning; state transitions and upgrade boundaries can make a local patch incomplete.
A score complement is not a misdiagnosis rate
The retrieved snapshot lists Opus 5.5 Max at 69.8%. The roughly 30% complement denotes unsuccessful tasks in this evaluation, not an established breakdown into missing evidence, reasoning errors or repair failures. The sample trajectory area reports no rubric data for the model, so task instructions cannot establish its actual tool behavior.
My interpretation is to use the benchmark to ask stage-specific questions: were necessary logs found, state transitions covered and verification conditions correct? Cross-study comparisons require matched tasks and judges; code-block coverage is not this task-success metric. Attributing failures requires trajectories and scoring evidence rather than inference from the aggregate.
Source: Original engineering article