Cloudflare’s code-review case shows that agent cost depends on more than model prices. Context organization, the number of calls required, and the useful findings those calls produce all matter. The team reports reducing average review cost from $2.43 to $0.77, approximately 68%. Quality changes differ across reviewer roles, however, and that distinction matters more than the headline percentage.
Original: Cutting our inference bill by 68% by being obsessive about optimization
Ryan Skidmore · First published: 2026-10-19
Why high cache hit rates can coexist with rising costs
Multi-turn reviews repeatedly carry earlier context. Even if most tokens are read from cache, growing context and more calls can still increase the bill. The article focuses on each session’s first request: frequent hits within a session are different from sharing a prefix across new sessions in different repositories.
Previously, shared instructions and environment details such as the working directory and date occupied one block, with a cache breakpoint at its end. Identical shared content therefore lacked its own reusable boundary. The change ends a block after stable content and places variable content afterward; preceding tool descriptions also need to remain consistent.
Reading the chart: gray shows before and orange after. Across all tiers, first attempts are compared by whether the first session request reused an earlier cache entry, not by whole-session token hit rate.
The transferable mechanism is to inspect the bytes actually sent to the provider. Identical prompt templates in source code do not establish identical rendered request prefixes. Investigating a similar problem calls for separating first and subsequent requests, shared-prefix length, and actual cached reads. A single aggregate token hit rate can hide a boundary-design problem. This is an engineering deduction from the case, not a requirement for every system to use the same prompt layout.
Shorter workflows also reduce repeated reads
The article describes removing unavailable tools, consolidating reads, skipping repeat reviews with no new relevant material, and stopping after publishing results. Their common effect is eliminating unnecessary later turns. A wasted call costs more than its own output: it may require another turn that carries and processes the existing conversation again.
This suggests measuring total cost per useful completed review. Cheaper individual LLM calls need not make a task cheaper if they cause additional retries. A somewhat larger initial input may pay for itself by eliminating round trips. Evaluation should record calls, input and output volumes, cache reads and writes, and task-completion state together.
Prefix sharing also has a concurrency trade-off. The article coordinates an initial cache write with subsequent reads by multiple reviewers. An engineering implication is that cache optimization can turn fully parallel startup into a short dependency chain. Whether waiting is worthwhile depends on saved computation, added delay, and failure propagation. Maximizing hit rate alone can obscure the review deadline.
Effort is not simply a performance switch
The team changes models and reasoning effort as well. The essential quality boundary is that code-quality review ultimately uses high effort, while performance review at medium effort loses approximately 64% of important findings according to the model judges. Security also shows a decline, without statistical significance. Lower aggregate cost does not establish preserved quality for every role.
Models and effort should therefore be treated as task strategies, rather than only resource settings. Roles differ in missed-defect cost, required investigation depth, and acceptable latency. A cheap configuration appropriate for formatting checks may be unsuitable for cross-file performance issues. Uniformly lowering effort can turn a local saving into a hard-to-detect capability loss.
This also explains why fewer findings alone do not establish regression. Removed findings might be duplicates, incorrect reports, or exaggerated severities; they might also be real problems. Comparing useful findings requires an account of what should have been detected, rather than just counting comments.
What the quality evidence establishes
Cloudflare examines whether authors edit flagged lines and conducts a blinded assessment of 657 findings. Two models from different providers judge findings, with a third resolving disagreements. These are quality signals for visible findings, not a complete measurement of missed issues. The article also notes groups containing only 35–66 findings and a latest observation window of one day.
An edit might indicate a useful report or an incidental change; no edit does not automatically mean the report was wrong. Judges given a diff can assess reported findings, but do not thereby establish a repository’s complete set of real defects. Support for high-effort code-quality review should therefore be understood as agreement among limited signals, not proof that every review task remains lossless.
Code and model-configuration changes also arrive at overlapping times. Per-change attribution uses before-and-after working-hour comparisons. The reported components are not a controlled experiment or an exact additive causal ledger. They are more useful for identifying major contributors and prioritizing subsequent experiments.
Observe cost, latency, and coverage together
The authors report reducing overall median review duration from 4m 09s to 1m 32s. Fewer round trips can improve both time and spending, but that alignment is not universal. Larger contexts, more parallel reviewers, or higher effort can change the relationship between quality and resource consumption.
An engineering approach is to proceed in two stages. First remove waste unrelated to task semantics, such as unavailable tools and work continuing after cancellation. Then compare models, effort, and skipping policies against quality references. The first category still needs behavioral checks; the second especially needs observation by role and risk tier rather than only global averages.
Skipping rules should also retain auditable reasons. A repeat review skipped because relevant files are unchanged differs from one skipped because a rule missed an indirect effect. Recording savings alongside trigger conditions makes that trade-off reconstructable after a missed finding. Likewise, cache savings should be understood together with which stable context was reused and whether it remains applicable to the task.
Read the cache-boundary and quality sections together. The advance is showing how to decompose an agent bill into understandable call structure and bring post-optimization capability changes back into the decision. Measurement practices and role-specific experiments transfer more readily than the 68% result, which combines changes in this particular system and period.